Hierarchical Point Cloud Compression

ABSTRACT

A system comprises an encoder configured to compress attribute information for a point cloud and/or a decoder configured to decompress compressed attribute information for the point cloud. To compress the attribute information, multiple levels of detail are generated based on an ordering of the points according to a space filling curve and attribute values are predicted. The attribute values may be predicted simultaneously while points are being assigned to different levels of detail. A decoder follows a similar prediction process based on level of details. Also, attribute correction values may be determined to correct predicted attribute values and may be used by a decoder to decompress a point cloud compressed using level of detail attribute compression. In some embodiments, attribute correction values may take into account an influence factor of respective points in a given level of detail on attributes in other levels of detail.

PRIORITY CLAIM

This application is a continuation of U.S. patent application Ser. No.16/508,202, filed Jul. 10, 2019 and entitled “Hierarchical Point CloudCompression”, which claims benefit of priority to the following U.S.Provisional Applications:

-   -   U.S. Provisional Application Ser. No. 62/696,295, filed Jul. 10,        2018;    -   U.S. Provisional Application Ser. No. 62/740,877, filed Oct. 3,        2018; and    -   U.S. Provisional Application Ser. No. 62/789,986, filed Jan. 8,        2019.        This application incorporates by reference the parent        application (U.S. patent application Ser. No. 16/508,202, filed        Jul. 10, 2019) and each of the above referenced provisional        applications to which the parent application claims priority, in        their entirety.

BACKGROUND Technical Field

This disclosure relates generally to compression and decompression ofpoint clouds comprising a plurality of points, each having associatedattribute information.

Description of the Related Art

Various types of sensors, such as light detection and ranging (LIDAR)systems, 3-D-cameras, 3-D scanners, etc. may capture data indicatingpositions of points in three dimensional space, for example positions inthe X, Y, and Z planes. Also, such systems may further capture attributeinformation in addition to spatial information for the respectivepoints, such as color information (e.g. RGB values), intensityattributes, reflectivity attributes, motion related attributes, modalityattributes, or various other attributes. In some circumstances,additional attributes may be assigned to the respective points, such asa time-stamp when the point was captured. Points captured by suchsensors may make up a “point cloud” comprising a set of points eachhaving associated spatial information and one or more associatedattributes. In some circumstances, a point cloud may include thousandsof points, hundreds of thousands of points, millions of points, or evenmore points. Also, in some circumstances, point clouds may be generated,for example in software, as opposed to being captured by one or moresensors. In either case, such point clouds may include large amounts ofdata and may be costly and time-consuming to store and transmit.

SUMMARY OF EMBODIMENTS

In some embodiments, a system includes one or more sensors configured tocapture points that collectively make up a point cloud, wherein each ofthe points comprises spatial information identifying a spatial locationof the respective point and attribute information defining one or moreattributes associated with the respective point. The system also includean encoder configured to compress the attribute information for thepoints. To compress the attribute information, the encoder is configuredto organize a points of the point cloud into an order according to aspace filling curve based on respective spatial positions of theplurality of points of the point cloud in 3D space. The encoder is alsoconfigured to assign an attribute value to at least one point of thepoint cloud based on the attribute information included in the capturedpoint cloud. Additionally, the encoder is configured to, for each ofrespective other ones of the points of the point cloud, identify a setof neighboring points, determine a predicted attribute value for therespective point based, at least in part, on predicted or assignedattributes values for the neighboring points, and determine, based, atleast in part, on comparing the predicted attribute value for therespective point to the attribute information for the point included inthe captured point cloud, an attribute correction value for the point.The encoder is configured to select points to be included in the one ormore additional levels of detail based, at least in part, on theirrespective positions in the order according to the space filling curve.Also, the encoder is configured to select neighboring points to use todetermine the predicted attribute value for the respective point forwhich an attribute value is being predicted based, at least in part, ontheir respective positions in the space filling relative to therespective point for which an attribute value is being predicted. Theencoder is further configured to encode the compressed attributeinformation for the point cloud, wherein the compressed attributeinformation comprises the assigned attribute value for the at least onepoint and data indicating, for the respective other ones of the points,the respective determined attribute correction values.

In some embodiments, a method comprises determining an order for aplurality of points of a point cloud according to a space filling curvebased on respective spatial positions of the points of the point cloudin 3D space. The method also comprises determining predicted attributevalues for points of the point cloud included in a first level of detailor one or more additional levels of detail based on neighboring pointsin a same level of detail as the point for which a predicted attributevalue is being determined, wherein points to be included in the firstlevel of detail and the one or more additional levels of detail areselected based, at least in part, on their respective positions in theorder according to the space filling curve, and wherein the neighboringpoints used to determine the predicted attribute value, for the pointfor which an attribute value is being predicted, are selected based, atleast in part, on their respective positions in the order according tothe space filling curve relative to the point for which an attributevalue is being predicted. Additionally, the method comprises determiningattribute correction values for the points of the point cloud includedin the first level of detail or the one or more additional levels ofdetail based on comparing the determined predicted attribute values forthe points to attribute values of corresponding points of the pointcloud. Furthermore, the method comprises applying an update operation tosmooth the attribute correction values, wherein the update operationtakes into account relative influences of the attributes of the pointsof a given level of detail on attribute values of points included inother levels of detail and encoding the updated attribute correctionvalues.

In some embodiments, a system includes a decoder configured to: receivecompressed attribute information for a point cloud comprising at leastone assigned attribute value for at least one point of the point cloudand data indicating, for other points of the point cloud, respectiveattribute correction values for respective attributes of the otherpoints. The decoder is further configured to, for each of respectiveother ones of the points of the point cloud other than the at least onepoint, identify a set of neighboring points to a point being evaluated,determine a predicted attribute value for the point being evaluatedbased, at least in part, on predicted or assigned attribute values forthe neighboring points, and adjust the predicted attribute value for thepoint being evaluated based, at least in part, on an attributecorrection value for the point included in the compressed attributeinformation. The decoder is configured to select the neighboring pointsused to determine the predicted attribute value for a point for which anattribute value is being predicted based, at least in part, on theirrespective positions in a space filling curve relative to the point forwhich an attribute value is being predicted. The decoder is furtherconfigured to provide attribute information for a decompressed pointcloud that is being reconstructed, the attribute information comprisingthe at least one assigned attribute value for the at least one point andthe adjusted predicted attribute values for the other ones of thepoints.

In some embodiments, a non-transitory computer-readable medium storesprogram instructions, that when executed on one or more processors,cause the one or more processors to implement an encoder as describedherein.

In some embodiments, a non-transitory computer-readable medium storesprogram instructions, that when executed on one or more processors,cause the one or more processors to implement a decoder as describedherein.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1A illustrates a system comprising a sensor that capturesinformation for points of a point cloud and an encoder that compressesattribute information and/or spatial information of the point cloud,where the compressed point cloud information is sent to a decoder,according to some embodiments.

FIG. 1B illustrates a process for encoding attribute information of apoint cloud, according to some embodiments.

FIG. 1C illustrates representative views of point cloud information atdifferent stages of an encoding process, according to some embodiments.

FIG. 2A illustrates components of an encoder, according to someembodiments.

FIG. 2B illustrates components of a decoder, according to someembodiments.

FIG. 3 illustrates an example compressed attribute file, according tosome embodiments.

FIG. 4A illustrates a process for compressing attribute information of apoint cloud, according to some embodiments.

FIG. 4B illustrates predicting attribute values as part of compressingattribute information of a point cloud using adaptive distance basedprediction, according to some embodiments.

FIGS. 4C-4E illustrate parameters that may be determined or selected byan encoder and signaled with compressed attribute information for apoint cloud, according to some embodiments.

FIG. 5 illustrates a process for encoding attribute correction values,according to some embodiments.

FIGS. 6A-B illustrate an example process for compressing spatialinformation of a point cloud, according to some embodiments.

FIG. 7 illustrates another example process for compressing spatialinformation of a point cloud, according to some embodiments.

FIG. 8 illustrates components an example encoder that generates ahierarchical level of detail (LOD) structure, according to someembodiments.

FIG. 9A illustrates an example level of detail (LOD) structure,according to some embodiments.

FIG. 9B illustrates an example compressed point cloud file comprisinglevel of details for a point cloud (LODs), according to someembodiments.

FIG. 10A illustrates an example process of encoding attribute valuesusing a bottom-up level of detail encoding process, according to someembodiments.

FIG. 10B illustrates an example process determining levels of detail,according to some embodiments.

FIG. 11 illustrates an example process of re-constructing attributevalues for a point cloud that was compressed using a bottom-up level ofdetail encoding process, according to some embodiments.

FIG. 12A illustrates a direct transformation that may be applied at anencoder to encode attribute information of a point could, according tosome embodiments.

FIG. 12B illustrates an inverse transformation that may be applied at adecoder to decode attribute information of a point cloud, according tosome embodiments.

FIG. 13 illustrates a key-word mapping process using a look-up tablethat may be used to compress updated attribute correction values,according to some embodiments.

FIG. 14 illustrates compressed point cloud information being used in a3-D telepresence application, according to some embodiments.

FIG. 15 illustrates compressed point cloud information being used in avirtual reality application, according to some embodiments.

FIG. 16 illustrates an example computer system that may implement anencoder or decoder, according to some embodiments.

This specification includes references to “one embodiment” or “anembodiment.” The appearances of the phrases “in one embodiment” or “inan embodiment” do not necessarily refer to the same embodiment.Particular features, structures, or characteristics may be combined inany suitable manner consistent with this disclosure.

“Comprising.” This term is open-ended. As used in the appended claims,this term does not foreclose additional structure or steps. Consider aclaim that recites: “An apparatus comprising one or more processor units. . . .” Such a claim does not foreclose the apparatus from includingadditional components (e.g., a network interface unit, graphicscircuitry, etc.).

“Configured To.” Various units, circuits, or other components may bedescribed or claimed as “configured to” perform a task or tasks. In suchcontexts, “configured to” is used to connote structure by indicatingthat the units/circuits/components include structure (e.g., circuitry)that performs those task or tasks during operation. As such, theunit/circuit/component can be said to be configured to perform the taskeven when the specified unit/circuit/component is not currentlyoperational (e.g., is not on). The units/circuits/components used withthe “configured to” language include hardware—for example, circuits,memory storing program instructions executable to implement theoperation, etc. Reciting that a unit/circuit/component is “configuredto” perform one or more tasks is expressly intended not to invoke 35U.S.C. § 112(f), for that unit/circuit/component. Additionally,“configured to” can include generic structure (e.g., generic circuitry)that is manipulated by software and/or firmware (e.g., an FPGA or ageneral-purpose processor executing software) to operate in manner thatis capable of performing the task(s) at issue. “Configure to” may alsoinclude adapting a manufacturing process (e.g., a semiconductorfabrication facility) to fabricate devices (e.g., integrated circuits)that are adapted to implement or perform one or more tasks.

“First,” “Second,” etc. As used herein, these terms are used as labelsfor nouns that they precede, and do not imply any type of ordering(e.g., spatial, temporal, logical, etc.). For example, a buffer circuitmay be described herein as performing write operations for “first” and“second” values. The terms “first” and “second” do not necessarily implythat the first value must be written before the second value.

“Based On.” As used herein, this term is used to describe one or morefactors that affect a determination. This term does not forecloseadditional factors that may affect a determination. That is, adetermination may be solely based on those factors or based, at least inpart, on those factors. Consider the phrase “determine A based on B.”While in this case, B is a factor that affects the determination of A,such a phrase does not foreclose the determination of A from also beingbased on C. In other instances, A may be determined based solely on B.

DETAILED DESCRIPTION

As data acquisition and display technologies have become more advanced,the ability to capture point clouds comprising thousands or millions ofpoints in 2-D or 3-D space, such as via LIDAR systems, has increased.Also, the development of advanced display technologies, such as virtualreality or augmented reality systems, has increased potential uses forpoint clouds. However, point cloud files are often very large and may becostly and time-consuming to store and transmit. For example,communication of point clouds over private or public networks, such asthe Internet, may require considerable amounts of time and/or networkresources, such that some uses of point cloud data, such as real-timeuses, may be limited. Also, storage requirements of point cloud filesmay consume a significant amount of storage capacity of devices storingthe point cloud files, which may also limit potential applications forusing point cloud data.

In some embodiments, an encoder may be used to generate a compressedpoint cloud to reduce costs and time associated with storing andtransmitting large point cloud files. In some embodiments, a system mayinclude an encoder that compresses attribute information and/or spatialinformation (also referred to herein as geometry information) of a pointcloud file such that the point cloud file may be stored and transmittedmore quickly than non-compressed point clouds and in a manner such thatthe point cloud file may occupy less storage space than non-compressedpoint clouds. In some embodiments, compression of spatial informationand/or attributes of points in a point cloud may enable a point cloud tobe communicated over a network in real-time or in near real-time. Forexample, a system may include a sensor that captures spatial informationand/or attribute information about points in an environment where thesensor is located, wherein the captured points and correspondingattributes make up a point cloud. The system may also include an encoderthat compresses the captured point cloud attribute information. Thecompressed attribute information of the point cloud may be sent over anetwork in real-time or near real-time to a decoder that decompressesthe compressed attribute information of the point cloud. Thedecompressed point cloud may be further processed, for example to make acontrol decision based on the surrounding environment at the location ofthe sensor. The control decision may then be communicated back to adevice at or near the location of the sensor, wherein the devicereceiving the control decision implements the control decision inreal-time or near real-time. In some embodiments, the decoder may beassociated with an augmented reality system and the decompressedattribute information may be displayed or otherwise used by theaugmented reality system. In some embodiments, compressed attributeinformation for a point cloud may be sent with compressed spatialinformation for points of the point cloud. In other embodiments, spatialinformation and attribute information may be separately encoded and/orseparately transmitted to a decoder.

In some embodiments, a system may include a decoder that receives one ormore point cloud files comprising compressed attribute information via anetwork from a remote server or other storage device that stores the oneor more point cloud files. For example, a 3-D display, a holographicdisplay, or a head-mounted display may be manipulated in real-time ornear real-time to show different portions of a virtual world representedby point clouds. In order to update the 3-D display, the holographicdisplay, or the head-mounted display, a system associated with thedecoder may request point cloud files from the remote server based onuser manipulations of the displays, and the point cloud files may betransmitted from the remote server to the decoder and decoded by thedecoder in real-time or near real-time. The displays may then be updatedwith updated point cloud data responsive to the user manipulations, suchas updated point attributes.

In some embodiments, a system, may include one or more LIDAR systems,3-D cameras, 3-D scanners, etc., and such sensor devices may capturespatial information, such as X, Y, and Z coordinates for points in aview of the sensor devices. In some embodiments, the spatial informationmay be relative to a local coordinate system or may be relative to aglobal coordinate system (for example, a Cartesian coordinate system mayhave a fixed reference point, such as a fixed point on the earth, or mayhave a non-fixed local reference point, such as a sensor location).

In some embodiments, such sensors may also capture attribute informationfor one or more points, such as color attributes, reflectivityattributes, velocity attributes, acceleration attributes, timeattributes, modalities, and/or various other attributes. In someembodiments, other sensors, in addition to LIDAR systems, 3-D cameras,3-D scanners, etc., may capture attribute information to be included ina point cloud. For example, in some embodiments, a gyroscope oraccelerometer, may capture motion information to be included in a pointcloud as an attribute associated with one or more points of the pointcloud.

In some embodiments, attribute information may comprise string values,such as different modalities. For example attribute information mayinclude string values indicating a modality such as “walking”,“running”, “driving”, etc. In some embodiments, an encoder may comprisea “string-value” to integer index, wherein certain strings areassociated with certain corresponding integer values. In someembodiments, a point cloud may indicate a string value for a point byincluding an integer associated with the string value as an attribute ofthe point. The encoder and decoder may both store a common string valueto integer index, such that the decoder can determine string values forpoints based on looking up the integer value of the string attribute ofthe point in a string value to integer index of the decoder that matchesor is similar to the string value to integer index of the encoder.

In some embodiments, an encoder compresses and encodes spatialinformation of a point cloud to compress the spatial information inaddition to compressing attribute information for attributes of thepoints of the point cloud. For example, to compress spatial informationa K-D tree may be generated wherein, respective numbers of pointsincluded in each of the cells of the K-D tree are encoded. This sequenceof encoded point counts may encode spatial information for points of apoint cloud. Also, in some embodiments, a sub-sampling and predictionmethod may be used to compress and encode spatial information for apoint cloud. In some embodiments, the spatial information may bequantized prior to being compressed and encoded. Also, in someembodiments, compression of spatial information may be lossless. Thus, adecoder may be able to determine a same view of the spatial informationas an encoder. Also, for lossy encoding, an encoder may be able todetermine a view of the spatial information a decoder will encounteronce the compressed spatial information is decoded. Because, both anencoder and decoder may have or be able to recreate the same spatialinformation for the point cloud, spatial relationships may be used tocompress attribute information for the point cloud.

For example, in many point clouds, attribute information betweenadjacent points or points that are located at relatively short distancesfrom each other may have high levels of correlation between attributes,and thus relatively small differences in point attribute values. Forexample, proximate points in a point cloud may have relatively smalldifferences in color, when considered relative to points in the pointcloud that are further apart.

In some embodiments, an encoder may include a predictor that determinesa predicted attribute value of an attribute of a point in a point cloudbased on attribute values for similar attributes of neighboring pointsin the point cloud and based on respective distances between the pointbeing evaluated and the neighboring points. In some embodiments,attribute values of attributes of neighboring points that are closer toa point being evaluated may be given a higher weighting than attributevalues of attributes of neighboring points that are further away fromthe point being evaluated. Also, the encoder may compare a predictedattribute value to an actual attribute value for an attribute of thepoint in the original point cloud prior to compression. A residualdifference, also referred to herein as an “attribute correction value”may be determined based on this comparison. An attribute correctionvalue may be encoded and included in compressed attribute informationfor the point cloud, wherein a decoder uses the encoded attributecorrection value to correct a predicted attribute value for the point,wherein the attribute value is predicted using a same or similarprediction methodology at the decoder that is the same or similar to theprediction methodology that was used at the encoder.

In some embodiments, to encode attribute values an encoder may generatean ordering of points of a point cloud based on spatial information forthe points of the point cloud. For example, the points may be orderedaccording a space-filling curve. In some embodiments, this ordering mayrepresent a Morton ordering of the points. The encoder may select afirst point as a starting point and may determine an evaluation orderfor other ones of the points of the point cloud based on minimumdistances from the starting point to a closest neighboring point, and asubsequent minimum distance from the neighboring point to the nextclosest neighboring point, etc. Also, in some embodiments, neighboringpoints may be determined from a sub-group of points within auser-defined search range of an index value of a given point beingevaluated, wherein the index value and the search range values arevalues in an index of the points of the point cloud organized accordingto the space filling curve. In this way, an evaluation order fordetermining predicted attribute values of the points of the point cloudmay be determined. Because the decoder may receive or re-create the samespatial information as the spatial information used by the encoder, thedecoder may generate the same ordering of the points for the point cloudand may determine the same evaluation order for the points of the pointcloud.

In some embodiments, an encoder may assign an attribute value for astarting point of a point cloud to be used to predict attribute valuesof other points of the point cloud. An encoder may predict an attributevalue for a neighboring point to the starting point based on theattribute value of the starting point and a distance between thestarting point and the neighboring point. The encoder may then determinea difference between the predicted attribute value for the neighboringpoint and the actual attribute value for the neighboring point includedin the non-compressed original point cloud. This difference may beencoded in a compressed attribute information file as an attributecorrection value for the neighboring point. The encoder may then repeata similar process for each point in the evaluation order. To predict theattribute value for subsequent points in the evaluation order, theencoder may identify the K-nearest neighboring points to a particularpoint being evaluated, wherein the identified K-nearest neighboringpoints have assigned or predicted attribute values. In some embodiments,“K” may be a configurable parameter that is communicated from an encoderto a decoder.

The encoder may determine a distance in X, Y, and Z space between apoint being evaluated and each of the identified neighboring points. Forexample, the encoder may determine respective Euclidian distances fromthe point being evaluated to each of the neighboring points. The encodermay then predict an attribute value for an attribute of the point beingevaluated based on the attribute values of the neighboring points,wherein the attribute values of the neighboring points are weightedaccording to an inverse of the distances from the point being evaluatedto the respective ones of the neighboring points. For example, attributevalues of neighboring points that are closer to the point beingevaluated may be given more weight than attribute values of neighboringpoints that are further away from the point being evaluated.

In a similar manner as described for the first neighboring point, theencoder may compare a predicted value for each of the other points ofthe point cloud to an actual attribute value in an originalnon-compressed point cloud, for example the captured point cloud. Thedifference may be encoded as an attribute correction value for anattribute of one of the other points that is being evaluated. In someembodiments, attribute correction values may be encoded in an order in acompressed attribute information file in accordance with the evaluationorder determined based on the space filling curve order. Because theencoder and the decoder may determine the same evaluation order based onthe spatial information for the point cloud, the decoder may determinewhich attribute correction value corresponds to which attribute of whichpoint based on the order in which the attribute correction values areencoded in the compressed attribute information file. Additionally, thestarting point and one or more attribute value(s) of the starting pointmay be explicitly encoded in a compressed attribute information filesuch that the decoder may determine the evaluation order starting withthe same point as was used to start the evaluation order at the encoder.Additionally, the one or more attribute value(s) of the starting pointmay provide a value of a neighboring point that a decoder uses todetermine a predicted attribute value for a point being evaluated thatis a neighboring point to the starting point.

In some embodiments, an encoder may determine a predicted value for anattribute of a point based on temporal considerations. For example, inaddition to or in place of determining a predicted value based onneighboring points in a same “frame” e.g. point in time as the pointbeing evaluated, the encoder may consider attribute values of the pointin adjacent and subsequent time frames.

FIG. 1A illustrates a system comprising a sensor that capturesinformation for points of a point cloud and an encoder that compressesattribute information of the point cloud, where the compressed attributeinformation is sent to a decoder, according to some embodiments.

System 100 includes sensor 102 and encoder 104. Sensor 102 captures apoint cloud 110 comprising points representing structure 106 in view 108of sensor 102. For example, in some embodiments, structure 106 may be amountain range, a building, a sign, an environment surrounding a street,or any other type of structure. In some embodiments, a captured pointcloud, such as captured point cloud 110, may include spatial andattribute information for the points included in the point cloud. Forexample, point A of captured point cloud 110 comprises X, Y, Zcoordinates and attributes 1, 2, and 3. In some embodiments, attributesof a point may include attributes such as R, G, B color values, avelocity at the point, an acceleration at the point, a reflectance ofthe structure at the point, a time stamp indicating when the point wascaptured, a string-value indicating a modality when the point wascaptured, for example “walking”, or other attributes. The captured pointcloud 110 may be provided to encoder 104, wherein encoder 104 generatesa compressed version of the point cloud (compressed attributeinformation 112) that is transmitted via network 114 to decoder 116. Insome embodiments, a compressed version of the point cloud, such ascompressed attribute information 112, may be included in a commoncompressed point cloud that also includes compressed spatial informationfor the points of the point cloud or, in some embodiments, compressedspatial information and compressed attribute information may becommunicated as separate files.

In some embodiments, encoder 104 may be integrated with sensor 102. Forexample, encoder 104 may be implemented in hardware or software includedin a sensor device, such as sensor 102. In other embodiments, encoder104 may be implemented on a separate computing device that is proximateto sensor 102.

FIG. 1B illustrates a process for encoding compressed attributeinformation of a point cloud, according to some embodiments. Also, FIG.1C illustrates representative views of point cloud information atdifferent stages of an encoding process, according to some embodiments.

At 152, an encoder, such as encoder 104, receives a captured point cloudor a generated point cloud. For example, in some embodiments a pointcloud may be captured via one or more sensors, such as sensor 102, ormay be generated in software, such as in a virtual reality or augmentedreality system. For example, 164 illustrates an example captured orgenerated point cloud. Each point in the point cloud shown in 164 mayhave one or more attributes associated with the point. Note that pointcloud 164 is shown in 2D for ease of illustration, but may includepoints in 3D space.

At 154, an ordering of the points of the point cloud is determinedaccording to a space filling curve. For example, a space filling curvemay fill a three dimensional space and points of a point cloud may beordered based on where they lie relative to the space filling curve. Forexample, a Morton code may be used to represent multi-dimensional datain one dimension, wherein a “Z-Order function” is applied to themultidimensional data to result in the one dimensional representation.In some embodiments, as discussed in more detail herein, the points mayalso be ordered into multiple levels of detail (LODs). In someembodiments, points to be included in respective levels of details(LODs) may be determined by ordering the points according to theirlocation along a space filling curve. For example, the points may beorganized according to their Morton codes.

In some embodiments, other space filling curves could be used. Forexample, techniques to map positions (e.g., in X, Y, Z coordinate form)to a space filling curve such as a Morton-order (or Z-order), Halbertcurve, Peano curve, and so on may be used. In this way all of the pointsof the point cloud that are encoded and decoded using the spatialinformation may be organized into an index in the same order on theencoder and the decoder. In order to determine various refinementlevels, sampling rates, etc. the ordered index of the points may beused. For example, to divide a point cloud into four levels of detail,an index that maps a Morton value to a corresponding point may besampled, for example at a rate of four, where every fourth indexed pointis included in the lowest level refinement. For each additional level ofrefinement remaining points in the index that have not yet been sampledmay be sampled, for example every third index point, etc. until all ofthe points are sampled for a highest level of detail

At 156, an attribute value for one or more attributes of a startingpoint may be assigned to be encoded and included in compressed attributeinformation for the point cloud. As discussed above, predicted attributevalues for points of a point cloud may be determined based on attributevalues of neighboring points. However, an initial attribute value for atleast one point is provided to a decoder so that the decoder maydetermine attribute values for other points using at least the initialattribute value and attribute correction values for correcting predictedattribute values that are predicted based on the initial attributevalue. Thus, one or more attribute values for at least one startingpoint are explicitly encoded in a compressed attribute information file.Additionally, spatial information for the starting point may beexplicitly encoded such that the starting point may be identified by adecoder to determine which point of the points of the point cloud is tobe used as a starting point for generating an order according to aspace-filling curve. In some embodiments, a starting point may beindicated in other ways other than explicitly encoding the spatialinformation for the starting point, such as flagging the starting pointor other methods of point identification.

Because a decoder will receive an indication of a starting point andwill encounter the same or similar spatial information for the points ofthe point cloud as the encoder, the decoder may determine a same spacefilling curve order from the same starting point as was determined bythe encoder. Additionally, the decoder may determine a same processingorder as the encoder based on the space filling curve order determinedby the decoder.

At 158, for a current point being evaluated, a prediction/correctionevaluator of an encoder determines a predicted attribute value for anattribute of the point currently being evaluated. In some embodiments, apoint currently being evaluated may have more than one attribute.Accordingly, a prediction/correction evaluator of an encoder may predictmore than one attribute value for the point. For each point beingevaluated, the prediction/correction evaluator may identify a set ofnearest neighboring points that have assigned or predicted attributevalues. In some embodiments, a number of neighboring points to identify,“K”, may be a configurable parameter of an encoder and the encoder mayinclude configuration information in a compressed attribute informationfile indicating the parameter “K” such that a decoder may identify asame number of neighboring points when performing attribute prediction.The prediction/correction evaluator may then determine distances betweenthe point being evaluated and respective ones of the identifiedneighboring points. The prediction/correction evaluator may use aninverse distance interpolation method to predict an attribute value foreach attribute of the point being evaluated. The prediction/correctionevaluator may then predict an attribute value of the point beingevaluated based on an average of inverse-distance weighted attributevalues of the identified neighboring points.

For example, 166 illustrates a point (X,Y,Z) being evaluated whereinattribute Al is being determined based on inverse distance weightedattribute values of eight identified neighboring points.

At 160, an attribute correction value is determined for each point. Theattribute correction value is determined based on comparing a predictedattribute value for each attribute of a point to corresponding attributevalues of the point in an original non-compressed point cloud, such asthe captured point cloud. For example, 168 illustrates an equation fordetermining attribute correction values, wherein a captured value issubtracted from a predicted value to determine an attribute correctionvalue. Note that while, FIG. 1B shows attribute values being predictedat 158 and attribute correction values being determined at 160, in someembodiments attribute correction values may be determined for a pointsubsequent to predicting an attribute value for the point. A next pointmay then be evaluated, wherein a predicted attribute value is determinedfor the point and an attribute correction value is determined for thepoint. Thus 158 and 160 may be repeated for each point being evaluated.In other embodiments, predicted values may be determined for multiplepoints and then attribute correction values may be determined. In someembodiments, predictions for subsequent points being evaluated may bebased on predicted attribute values or may be based on correctedattribute values or both. In some embodiments, both an encoder and adecoder may follow the same rules as to whether predicted values forsubsequent points are to be determined based on predicted or correctedattribute values.

At 162, the determined attribute correction values for the points of thepoint cloud, one or more assigned attribute values for the startingpoint, spatial information or other indicia of the starting point, andany configuration information to be included in a compressed attributeinformation file is encoded. As discussed in more detail in FIG. 5various encoding methods, such as arithmetic encoding and/or Golombencoding may be used to encode the attribute correction values, assignedattribute values, and the configuration information.

FIG. 2A illustrates components of an encoder, according to someembodiments.

Encoder 202 may be a similar encoder as encoder 104 illustrated in FIG.1A. Encoder 202 includes spatial encoder 204, space filling curve ordergenerator 210, prediction/correction evaluator 206, incoming datainterface 214, and outgoing data interface 208. Encoder 202 alsoincludes context store 216 and configuration store 218.

In some embodiments, a spatial encoder, such as spatial encoder 204, maycompress spatial information associated with points of a point cloud,such that the spatial information can be stored or transmitted in acompressed format. In some embodiments, a spatial encoder, may utilizeK-D trees to compress spatial information for points of a point cloud asdiscussed in more detail in regard to FIG. 7. Also, in some embodiments,a spatial encoder, such as spatial encoder 204, may utilize asub-sampling and prediction technique as discussed in more detail inregard to FIGS. 6A-B. In some embodiments, a spatial encoder, such asspatial encoder 204, may utilize Octrees to compress spatial informationfor points of a point cloud or various other techniques to compressionspatial information for points of a point cloud.

In some embodiments, compressed spatial information may be stored ortransmitted with compressed attribute information or may be stored ortransmitted separately. In either case, a decoder receiving compressedattribute information for points of a point cloud may also receivecompressed spatial information for the points of the point cloud, or mayotherwise obtain the spatial information for the points of the pointcloud.

A space filling curve order generator, such as space filling curve ordergenerator 210, may utilize spatial information for points of a pointcloud to generate an indexed order of the points based on where thepoints fall along a space filling curve. For example Morton codes may begenerated for the points of the point cloud. Because a decoder isprovided or otherwise obtains the same spatial information for points ofa point cloud as are available at the encoder, a space filling curveorder determined by a space filling curve order generator of an encoder,such as space filling curve order generator 210 of encoder 202, may bethe same or similar as a space filling curve order generated by a spacefilling curve order generator of a decoder, such as space filling curveorder generator 228 of decoder 220.

A prediction/correction evaluator, such as prediction/correctionevaluator 206 of encoder 202, may determine predicted attribute valuesfor points of a point cloud based on an inverse distance interpolationmethod using attribute values of the K-nearest neighboring points of apoint for whom an attribute value is being predicted. Theprediction/correction evaluator may also compare a predicted attributevalue of a point being evaluated to an original attribute value of thepoint in a non-compressed point cloud to determine an attributecorrection value. In some embodiments, a prediction/correctionevaluator, such as prediction/correction evaluator 206 of encoder, 202may adaptively adjust a prediction strategy used to predict attributevalues of points in a given neighborhood of points based on ameasurement of the variability of the attribute values of the points inthe neighborhood.

An outgoing data encoder, such as outgoing data encoder 208 of encoder202, may encode attribute correction values and assigned attributevalues included in a compressed attribute information file for a pointcloud. In some embodiments, an outgoing data encoder, such as outgoingdata encoder 208, may select an encoding context for encoding a value,such as an assigned attribute value or an attribute correction value,based on a number of symbols included in the value. In some embodiments,values with more symbols may be encoded using an encoding contextcomprising Golomb exponential encoding, whereas values with fewersymbols may be encoded using arithmetic encoding. In some embodiments,encoding contexts may include more than one encoding technique. Forexample, a portion of a value may be encoded using arithmetic encodingwhile another portion of the value may be encoded using Golombexponential encoding. In some embodiments, an encoder, such as encoder202, may include a context store, such as context store 216, that storesencoding contexts used by an outgoing data encoder, such as outgoingdata encoder 208, to encode attribute correction values and assignedattribute values.

In some embodiments, an encoder, such as encoder 202, may also includean incoming data interface, such as incoming data interface 214. In someembodiments, an encoder may receive incoming data from one or moresensors that capture points of a point cloud or that capture attributeinformation to be associated with points of a point cloud. For example,in some embodiments, an encoder may receive data from an LIDAR system,3-D-camera, 3-D scanner, etc. and may also receive data from othersensors, such as a gyroscope, accelerometer, etc. Additionally, anencoder may receive other data such as a current time from a systemclock, etc. In some embodiments, such different types of data may bereceived by an encoder via an incoming data interface, such as incomingdata interface 214 of encoder 202.

In some embodiments, an encoder, such as encoder 202, may furtherinclude a configuration interface, such as configuration interface 212,wherein one or more parameters used by the encoder to compress a pointcloud may be adjusted via the configuration interface. In someembodiments, a configuration interface, such as configuration interface212, may be a programmatic interface, such as an API. Configurationsused by an encoder, such as encoder 202, may be stored in aconfiguration store, such as configuration store 218.

In some embodiments, an encoder, such as encoder 202, may include moreor fewer components than shown in FIG. 2A.

FIG. 2B illustrates components of a decoder, according to someembodiments.

Decoder 220 may be a similar decoder as decoder 116 illustrated in FIG.1A. Decoder 220 includes encoded data interface 226, spatial decoder222, space filling curve order generator 228, prediction evaluator 224,context store 232, configuration store 234, and decoded data interface220.

A decoder, such as decoder 220, may receive an encoded compressed pointcloud and/or an encoded compressed attribute information file for pointsof a point cloud. For example, a decoder, such as decoder 220, mayreceive a compressed attribute information file, such a compressedattribute information 112 illustrated in FIG. 1A or compressed attributeinformation file 300 illustrated in FIG. 3. The compressed attributeinformation file may be received by a decoder via an encoded datainterface, such as encoded data interface 226. The encoded compressedpoint cloud may be used by the decoder to determine spatial informationfor points of the point cloud. For example, spatial information ofpoints of a point cloud included in a compressed point cloud may begenerated by a spatial information generator, such as spatialinformation generator 222. In some embodiments, a compressed point cloudmay be received via an encoded data interface, such as encoded datainterface 226, from a storage device or other intermediary source,wherein the compressed point cloud was previously encoded by an encoder,such as encoder 104.

In some embodiments, an encoded data interface, such as encoded datainterface 226, may decode spatial information. For example the spatialinformation may have been encoded using various encoding techniques suchas arithmetic encoding, Golomb encoding, etc. A spatial informationgenerator, such as spatial information generator 222, may receivedecoded spatial information from an encoded data interface, such asencoded data interface 226, and may use the decoded spatial informationto generate a representation of the geometry of the point cloud beingde-compressed. For example, decoded spatial information may be formattedas residual values to be used in a sub-sampled prediction method torecreate a geometry of a point cloud to be decompressed. In suchsituations, the spatial information generator 222, may recreate thegeometry of the point cloud being decompressed using decoded spatialinformation from encoded data interface 226, and space filling curveorder generator 228 may determine a space filling curve order for thepoint cloud being decompressed based on the recreated geometry for thepoint cloud being decompressed generated by spatial informationgenerator 222.

Once spatial information for a point cloud is determined and aspace-filling curve order has been determined, the space-filling curveorder may be used by a prediction evaluator of a decoder, such asprediction evaluator 224 of decoder 220, to determine an evaluationorder for determining attribute values of points of the point cloud.Additionally, the space-filling curve order may be used by a predictionevaluator, such as prediction evaluator 224, to identify nearestneighboring points to a point being evaluated.

A prediction evaluator of a decoder, such as prediction evaluator 224,may select a starting point of a s based on an assigned starting pointincluded in a compressed attribute information file. In someembodiments, the compressed attribute information file may include oneor more assigned values for one or more corresponding attributes of thestarting point. In some embodiments, a prediction evaluator, such asprediction evaluator 224, may assign values to one or more attributes ofa starting point in a decompressed model of a point cloud beingdecompressed based on assigned values for the starting point included ina compressed attribute information file. A prediction evaluator, such asprediction evaluator 224, may further utilize the assigned values of theattributes of the starting point to determine attribute values ofneighboring points. For example, a prediction evaluator may select aneighboring point to the starting point as a next point to evaluate,wherein the neighboring point is selected based on an index order of thepoints according to the space-filling curve order. Note that because thespace-filling curve order is generated based on the same or similarspatial information at the decoder as was used to generate aspace-filling curve order at an encoder, the decoder may determine thesame evaluation order for evaluating the points of the point cloud beingdecompressed as was determined at the encoder by identifying nextnearest neighbors in an index according to the space-filling curveorder.

Once the prediction evaluator has identified the “K” nearest neighboringpoints to a point being evaluated, the prediction evaluator may predictone or more attribute values for one or more attributes of the pointbeing evaluated based on attribute values of corresponding attributes ofthe “K” nearest neighboring points. In some embodiments, an inversedistance interpolation technique may be used to predict an attributevalue of a point being evaluated based on attribute values ofneighboring points, wherein attribute values of neighboring points thatare at a closer distance to the point being evaluated are weighted moreheavily than attribute values of neighboring points that are at furtherdistances from the point being evaluated. In some embodiments, aprediction evaluator of a decoder, such as prediction evaluator 224 ofdecoder 220, may adaptively adjust a prediction strategy used to predictattribute values of points in a given neighborhood of points based on ameasurement of the variability of the attribute values of the points inthe neighborhood. For example, in embodiments wherein adaptiveprediction is used, the decoder may mirror prediction adaptationdecisions that were made at an encoder. In some embodiments, adaptiveprediction parameters may be included in compressed attributeinformation received by the decoder, wherein the parameters weresignaled by an encoder that generated the compressed attributeinformation. In some embodiments, a decoder may utilize one or moredefault parameters in the absence of a signaled parameter, or may inferparameters based on the received compressed attribute information.

A prediction evaluator, such as prediction evaluator 224, may apply anattribute correction value to a predicted attribute value to determinean attribute value to include for the point in a decompressed pointcloud. In some embodiments, an attribute correction value for anattribute of a point may be included in a compressed attributeinformation file. In some embodiments, attribute correction values maybe encoded using one of a plurality of supported coding contexts,wherein different coding contexts are selected to encode differentattribute correction values based on a number of symbols included in theattribute correction value. In some embodiments, a decoder, such asdecoder 220, may include a context store, such as context store 232,wherein the context store stores a plurality of encoding context thatmay be used to decode assigned attribute values or attribute correctionvalues that have been encoded using corresponding encoding contexts atan encoder.

A decoder, such as decoder 220, may provide a decompressed point cloudgenerated based on a received compressed point cloud and/or a receivedcompressed attribute information file to a receiving device orapplication via a decoded data interface, such as decoded data interface230. The decompressed point cloud may include the points of the pointcloud and attribute values for attributes of the points of the pointcloud. In some embodiments, a decoder may decode some attribute valuesfor attributes of a point cloud without decoding other attribute valuesfor other attributes of a point cloud. For example, a point cloud mayinclude color attributes for points of the point cloud and may alsoinclude other attributes for the points of the point cloud, such asvelocity, for example. In such a situation, a decoder may decode one ormore attributes of the points of the point cloud, such as the velocityattribute, without decoding other attributes of the points of the pointcloud, such as the color attributes.

In some embodiments, the decompressed point cloud and/or decompressedattribute information file may be used to generate a visual display,such as for a head mounted display. Also, in some embodiments, thedecompressed point cloud and/or decompressed attribute information filemay be provided to a decision making engine that uses the decompressedpoint cloud and/or decompressed attribute information file to make oneor more control decisions. In some embodiments, the decompressed pointcloud and/or decompressed attribute information file may be used invarious other applications or for various other purposes.

FIG. 3 illustrates an example compressed attribute information file,according to some embodiments. Attribute information file 300 includesconfiguration information 302, point cloud data 304, and point attributecorrection values 306. In some embodiments, point cloud file 300 may becommunicated in parts via multiple packets. In some embodiments, not allof the sections shown in attribute information file 300 may be includedin each packet transmitting compressed attribute information. In someembodiments, an attribute information file, such as attributeinformation file 300, may be stored in a storage device, such as aserver that implements an encoder or decoder, or other computing device.In some embodiments, additional configuration information may includeadaptive prediction parameters, such as a variability measurementtechnique to use to determine a variability measurement for aneighborhood of points, a threshold variability value to trigger use ofa particular prediction procedure, one or more parameters fordetermining a size of a neighborhood of points for which variability isto be determined, etc.

FIG. 4A illustrates a process for compressing attribute information of apoint cloud, according to some embodiments.

At 402, an encoder receives a point cloud that includes attributeinformation for at least some of the points of the point cloud. Thepoint cloud may be received from one or more sensors that capture thepoint cloud, or the point cloud may be generated in software. Forexample, a virtual reality or augmented reality system may havegenerated the point cloud.

At 404, the spatial information of the point cloud, for example X, Y,and Z coordinates for the points of the point cloud may be quantized. Insome embodiments, coordinates may be rounded off to the nearestmeasurement unit, such as a meter, centimeter, millimeter, etc.

At 406, the quantized spatial information is compressed. In someembodiments, spatial information may be compressed using a sub-samplingand subdivision prediction technique as discussed in more detail inregard to FIGS. 6A-B. Also, in some embodiments, spatial information maybe compressed using a K-D tree compression technique as discussed inmore detail in regard to FIG. 7, or may be compressed using an Octreecompression technique. In some embodiments, other suitable compressiontechniques may be used to compress spatial information of a point cloud.

At 408, the compressed spatial information for the point cloud isencoded as a compressed point cloud file or a portion of a compressedpoint cloud file. In some embodiments, compressed spatial informationand compressed attribute information may be included in a commoncompressed point cloud file, or may be communicated or stored asseparate files.

At 412, the received spatial information of the point cloud is used togenerate an indexed point order according to a space-filling curve. Insome embodiments, the spatial information of the point cloud may bequantized before generating the order according to the space-fillingcurve. Additionally, in some embodiments wherein a lossy compressiontechnique is used to compress the spatial information of the pointcloud, the spatial information may be lossy encoded and lossy decodedprior to generating the order according to the space filling curve. Inembodiments that utilize lossy compression for spatial information,encoding and decoding the spatial information at the encoder may ensurethat an order according to a space filling curve generated at theencoder will match an order according to the space filling curve thatwill be generated at a decoder using decoded spatial information thatwas previously lossy encoded.

Additionally, in some embodiments, at 410, attribute information forpoints of the point cloud may be quantized. For example attribute valuesmay be rounded to whole numbers or to particular measurement increments.In some embodiments wherein attribute values are integers, such as whenintegers are used to communicate string values, such as “walking”,“running”, “driving”, etc., quantization at 410 may be omitted.

At 414, attribute values for a starting point are assigned. The assignedattribute values for the starting point are encoded in a compressedattribute information file along with attribute correction values.Because a decoder predicts attribute values based on distances toneighboring points and attribute values of neighboring points, at leastone attribute value for at least one point is explicitly encoded in acompressed attribute file. In some embodiments, points of a point cloudmay comprise multiple attributes and at least one attribute value foreach type of attribute may be encoded for at least one point of thepoint cloud, in such embodiments. In some embodiments, a starting pointmay be a first point evaluated when determining the order according tothe space filling curve at 412. In some embodiments, an encoder mayencode data indicating spatial information for a starting point and/orother indicia of which point of the point cloud is the starting point orstarting points. Additionally, the encoder may encode attribute valuesfor one or more attributes of the starting point.

At 416, the encoder determines an evaluation order for predictingattribute values for other points of the point cloud, other than thestarting point, said predicting and determining attribute correctionvalues, may be referred to herein as “evaluating” attributes of a point.The evaluation order may be determined based on the order according tothe space filling curve.

At 418, a neighboring point of the starting point or of a subsequentpoint being evaluated is selected. In some embodiments, a neighboringpoint to be next evaluated may be selected based on the neighboringpoint being a next point in an indexed order of points according to aspace filling curve.

At 420, the “K” nearest neighboring points to the point currently beingevaluated are determined. The parameter “K” may be a configurableparameter selected by an encoder or provided to an encoder as a userconfigurable parameter. In order to select the “K” nearest neighboringpoints, an encoder may identify the first “K” nearest points to a pointbeing evaluated according to the indexed order of points determined at412 and respective distances between the points. For example, instead ofdetermining the absolute nearest neighboring points to a point beingevaluated, an encoder may select a sub-group of points of the pointcloud having index values in the index according to the space-fillingcurve that are within a user defined search range, e.g. 8, 16, 32, 64,etc. of an index value of a particular point being evaluated. Theencoder may then utilize distances within the sub-group of points toselect the “K” nearest neighboring points to use for prediction. In someembodiments, only points having assigned attribute values or for whichpredicted attribute values have already been determined may be includedin the “K” nearest neighboring points. In some embodiments variousnumbers of points may be identified. For example, in some embodiments,“K” may be 5 points, 10 points, 16 points, etc. Because a point cloudcomprises points in 3-D space a particular point may have multipleneighboring points in multiple planes. In some embodiments, an encoderand a decoder may be configured to identify points as the “K” nearestneighboring points regardless of whether or not a value has already beenpredicted for the point. Also, in some embodiments, attribute values forpoints used in predication may be previously predicted attribute valuesor corrected predicted attribute values that have been corrected basedon applying an attribute correction value. In either case, an encoderand a decoder may be configured to apply the same rules when identifyingthe “K” nearest neighboring points and when predicting an attributevalue of a point based on attribute values of the “K” nearestneighboring points.

At 422, one or more attribute values are determined for each attributeof the point currently being evaluated. The attribute values may bedetermined based on an inverse distance interpolation. The inversedistance interpolation may interpolate the predicted attribute valuebased on the attribute values of the “K” nearest neighboring points. Theattribute values of the “K” nearest neighboring points may be weightedbased on respective distances between respective ones of the “K” nearestneighboring points and the point being evaluated. Attribute values ofneighboring points that are at shorter distances from the pointcurrently being evaluated may be weighted more heavily than attributevalues of neighboring points that are at greater distances from thepoint currently being evaluated.

At 424, attribute correction values are determined for the one or morepredicted attribute values for the point currently being evaluated. Theattribute correction values may be determined based on comparing thepredicted attribute values to corresponding attribute values for thesame point (or a similar point) in the point cloud prior to attributeinformation compression. In some embodiments, quantized attributeinformation, such as the quantized attribute information generated at410, may be used to determine attribute correction values. In someembodiments, an attribute correction value may also be referred to as a“residual error” wherein the residual error indicates a differencebetween a predicted attribute value and an actual attribute value.

At 426, it is determined if there are additional points in the pointcloud for which attribute correction values are to be determined. Ifthere are additional points to evaluate, the process reverts to 418 andthe next point in the evaluation order is selected to be evaluated. Theprocess may repeat steps 418-426 until all or a portion of all of thepoints of the point cloud have been evaluated to determine predictedattribute values and attribute correction values for the predictedattribute values.

At 428, the determined attribute correction values, the assignedattribute values, and any configuration information for decoding thecompressed attribute information file, such as a parameter “K”, isencoded.

Adaptive Attribute Prediction

In some embodiments, an encoder as described above may furtheradaptively change a prediction strategy and/or a number of points usedin a given prediction strategy based on attribute values of neighboringpoints. Also, a decoder may similarly adaptively change a predictionstrategy and/or a number of points used in a given prediction strategybased on reconstructed attribute values of neighboring points.

For example, a point cloud may include points representing a road wherethe road is black with a white stripe on the road. A default nearestneighbor prediction strategy may be adaptively changed to take intoaccount the variability of attribute values for points representing thewhite line and the black road. Because these points have a largedifference in attribute values, a default nearest neighbor predictionstrategy may result in blurring of the white line and/or high residualvalues that decrease a compression efficiency. However, an updatedprediction strategy may account for this variability by selecting abetter suited prediction strategy and/or by using less points in aK-nearest neighbor prediction. For example, for the black road, notusing the white line points in a K-nearest neighbor prediction.

In some embodiments, before predicting an attribute value for a point P,an encoder or decoder may compute the variability of attribute values ofpoints in a neighborhood of point P, for example the K-nearestneighboring points. In some embodiments, variability may be computedbased on a variance, a maximum difference between any two attributevalues (or reconstructed attribute values) of the points neighboringpoint P. In some embodiments, variability may be computed based on aweighted average of the neighboring points, wherein the weighted averageaccounts for distances of the neighboring points to point P. In someembodiments, variability for a group of neighboring points may becomputed based on a weighted averages for attributes for the neighboringpoints and taking into account distances to the neighboring points. Forexample,

Variability=E[(X−weighted mean(X))²]

In the above equation, E is the mean attribute value of the points inthe neighborhood of point P, the weighted mean(X) is a weighted mean ofthe attribute values of the points in the neighborhood of point P thattakes into account the distances of the neighboring points from point P.In some embodiments, the variability may be calculated as the maximumdifference compared to the mean value of the attributes, E(X), theweighted mean of the attributes, weighted mean(X), or the median valueof the attributes, median(X). In some embodiments, the variability maybe calculated using the average of the values corresponding to the xpercent, e.g. x=10 that have the largest difference as compared to themean value of the attributes, E(X), the weighted mean of the attributes,weighted mean(X), or the median value of the attributes, median(X).

In some embodiments, if the calculated variability of the attributes ofthe points in the neighborhood of point P is greater than a thresholdvalue, then a rate-distortion optimization may be applied. For example,a rate-distortion optimization may reduce a number of neighboring pointsused in a prediction or switch to a different prediction technique. Insome embodiments, the threshold may be explicitly written in thebit-stream. Also, in some embodiments, the threshold may be adaptivelyadjusted per point cloud, or sub-block of the point cloud or for anumber of points to be encoded. For example, a threshold may be includedin compressed attribute information file 350 as additional configurationinformation included in configuration information 302, as described inFIG. 3, or may be included in compressed attribute file 950 asadditional configuration information included in configurationinformation 952, as described below in regard to FIG. 9B.

In some embodiments, different distortion measures may be used in arate-distortion optimization procedure, such as sum of squares error,weighted sum of squares error, sum of absolute differences, or weightedsum of absolute differences.

In some embodiments, distortion could be computed independently for eachattribute, or multiple attributes corresponding to the same sample andcould be considered, and appropriately weighted. For example, distortionvalues for R, G, B or Y, U, V could be computed and then combinedtogether linearly or non-linearly to generate an overall distortionvalue.

In some embodiments, advanced techniques for rate distortionquantization, such as trellis based quantization could also beconsidered where, instead of considering a single point in isolationmultiple points are coded jointly. The coding process, for example, mayselect to encode all these multiple points using the method that resultsin minimizing a cost function of the form J=D+lambda*Rate, where D isthe overall distortion for all these points, and Rate is the overallrate cost for coding these points.

In some embodiments, an encoder, such as encoder 202, may explicitlyencode an index value of a chosen prediction strategy for a point cloud,for a level of detail of a point cloud, or for a group of points withina level of detail of a point cloud, wherein the decoder has access to aninstance of the index and can determine the chosen prediction strategybased on the received index value. The decoder may apply the chosenprediction strategy for the set of points for which the rate-distortionoptimization procedure is being applied. In some embodiments, there maybe a default prediction strategy and the decoder may apply the defaultprediction strategy if no rate-distortion optimization procedure isspecified in the encoded bit stream. Also, in some embodiments a defaultprediction strategy may be applied if no variability threshold is met.

For example, FIG. 4B illustrates predicting attribute values as part ofcompressing attribute information of a point cloud using adaptivedistance based prediction, according to some embodiments.

In some embodiments in which adaptive distance based prediction isemployed, predicting attribute values as described in elements 420 and422 of FIG. 4A may further include steps such as 450-456 to select aprediction procedure to be used to predict the attribute values for thepoints. In some embodiments the selected prediction procedure may be aK-nearest neighbor prediction procedure, as described herein and inregard to element 420 in FIG. 4A. In some embodiments, the selectedprediction procedure may be a modified K-nearest neighbor predictionprocedure, wherein fewer points are included in the number of nearestneighbors used to perform the adaptive prediction than a number ofpoints used to predict attribute values for portions of the point cloudwith less variability. In some embodiments, the selected predictionprocedure may be that the point for which an attribute value is beingpredicted simply uses the attribute value of the nearest point to thepoint for which the attribute value is being predicted, if thevariability of the neighboring points exceeds a threshold associatedwith this prediction procedure. In some embodiments, other predictionprocedures may be used depending on the variability of points in aneighborhood of a point for which an attribute value is being predicted.For example, in some embodiments, other prediction procedures, such as anon-distance based interpolation procedure may be used, such asbarycentric interpolation, natural neighbor interpolation, moving leastsquares interpolation, or other suitable interpolation techniques.

At 450, the encoder identifies a set of neighboring points for aneighborhood of a point of the point cloud for which an attribute valueis being predicted. In some embodiments, the set of neighboring pointsof the neighborhood may be identified using a K-nearest neighbortechnique as described herein. In some embodiments, points to be used todetermine variability may be identified in other manners. For example,in some embodiments, a neighborhood of points used for variabilityanalysis may be defined to include more or fewer points or points withina greater or smaller distance from the given point than are used topredict attribute values based on inverse distance based interpolationusing the K-nearest neighboring points. In some embodiments, whereinparameters used to identify the neighborhood points for determiningvariability differ from the parameters used in a K-nearest neighborprediction, the differing parameters or data from which the differingparameter may be determined is signaled in a bit stream encoded by theencoder.

At 452, the variability of the attribute values of the neighboringpoints is determined. In some embodiment, each attribute valuevariability may be determined separately. For example, for points withR, G, B attribute values each attribute value (e.g. each of R, G, and B)may have their respective variabilities determined separately. Also, insome embodiments trellis quantization may be used wherein a set ofattributes such as RGB that have correlated values may be determined asa common variability. For example, in the example discussed above withregard to the white stripe on the black road, the large variability in Rmay also apply to B and G, thus it is not necessary to determinevariability for each of R, G, and B separately. Instead the relatedattribute values can be considered as a group and a common variabilityfor the correlated attributes can be determined.

In some embodiments, the variability of the attributes in theneighborhood of point P may be determined using: a sum of square errorsvariability technique, a distance weighted sum of square errorsvariability technique, a sum of absolute differences variabilitytechnique, a distance weighted sum of absolute differences variabilitytechnique, or other suitable variability technique. In some embodimentsthe encoder may select a variability technique to be used for a givenpoint P, and may encode in a bit stream encoded by the encoder an indexvalue for an index of variability techniques, wherein the decoderincludes the same index and can determine which variability technique touse for point P based on the encoded index value.

At 454 through 456 it is determined whether or not the variabilitydetermined at 452 exceeds one or more variability thresholds. If so, acorresponding prediction technique that corresponds with the exceededvariability threshold is used to predict the attribute value or valuesfor the point P. In some embodiments, multiple prediction procedures maybe supported. For example, element 458 indicates using a firstprediction procedure if a first variability threshold is exceeded andelement 460 indicates using another prediction procedure if anothervariability threshold is exceeded. Furthermore, 462 indicates using adefault prediction procedure, such as a non-modified K-nearest neighborprediction procedure if the variability thresholds 1 through N are notexceeded. In some embodiments, a single variability threshold and asingle alternate prediction procedure may be used in addition to adefault prediction procedure. In some embodiments, any number of “N”variability thresholds and corresponding prediction procedures may beused.

For example, in some embodiments, if a first variability threshold isexceeded a first prediction procedure may be to use fewer neighboringpoints than are used in the default K-nearest neighbor predictionprocedure. Also, if a second variability threshold is exceeded, a secondprediction procedure may be to use only the nearest point to determinethe attribute value of the point P. Thus, in such embodiments, mediumvariability may cause some outlier points to be omitted under the firstprediction procedure and higher variability may cause all but theclosest neighboring point to be omitted from the prediction procedure,while if variability is low, the K-nearest neighboring points are usedin the default prediction procedure.

FIGS. 4C-4E illustrate parameters that may be determined or selected byan encoder and signaled with compressed attribute information for apoint cloud, according to some embodiments.

In FIG. 4C at 470, an encoder may select a variability measurementtechnique to be used to determine attribute variability for points in aneighborhood of a point P for which an attribute value is beingpredicted. In some embodiments, the encoder may utilize a ratedistortion optimization framework to determine which variabilitymeasurement technique to use. At 472 the encoder may include, in a bitstream encoded by the encoder, a signal indicating which variabilitytechnique was selected.

In FIG. 4D at 480, an encoder may determine a variability threshold forpoints in a neighborhood of a point P for which an attribute value isbeing predicted. In some embodiments, the encoder may utilize a ratedistortion optimization framework to determine the variabilitythreshold. At 482 the encoder may include in a bit stream, encoded bythe encoder, a signal indicating which variability threshold was used bythe encoder to perform prediction.

In FIG. 4E at 490, an encoder may determine or select a neighborhoodsize for use in determining variability. For example, the encoder mayuse a rate distortion optimization technique to determine how big orsmall of a neighborhood of points to use in determining variability forpoint P. At 492, the encoder may include in a bit stream, encoded by theencoder, one or more values for defining the neighborhood size. Forexample, the encoder may signal a minimum distance from point P, amaximum distance from point P, a total number of neighboring points toinclude, etc. and these parameters may define which points are includedin the neighborhood points for point P that are considered indetermining variability.

In some embodiments, one or more of the variability technique,variability threshold, or neighborhood size may not be signaled and mayinstead be determined at a decoder using a pre-determined parameterknown to both the encoder and decoder. In some embodiments, a decodermay infer one or more of the variability technique, variabilitythreshold, or neighborhood size to be used based on other data, such asspatial information for the point cloud.

Once the attribute values are predicted using the appropriatecorresponding prediction procedure at 458-462, the decoder may proceedto apply attribute correction values received in the encoded bit streamto adjust the predicted attribute values. In some embodiments, usingadaptive prediction as described herein at the encoder and decoder mayreduce a number of bits necessary to encode the attribute correctionvalues and may also reduce distortion of a re-constructed point cloudre-constructed at the decoder using the prediction procedures and thesignaled attribute correction values.

Example Process for Encoding Attribute Values and/or AttributeCorrection Values

The attribute correction values, the assigned attribute values, and anyconfiguration information may be encoded using various encodingtechniques.

For example, FIG. 5 illustrates a process for encoding attributecorrection values, according to some embodiments. At 502, an attributecorrection value for a point whose values (e.g. attribute correctionvalues) are being encoded is converted to an unsigned value. Forexample, in some embodiments, attribute correction values that arenegative values may be assigned odd numbers and attribute correctionvalues that are positive values may be assigned even numbers. Thus,whether or not the attribute correction value is positive or negativemay be implied based on whether or not a value of the attributecorrection value is an even number or an odd number. In someembodiments, assigned attribute values may also be converted intounsigned values. In some embodiments, attribute values may all bepositive values, for example in the case of integers that are assignedto represent string values, such as “walking”, “running”, “driving” etc.In such cases, 502 may be omitted.

At 504, an encoding context is selected for encoding a first value for apoint. The value may be an assigned attribute value or may be anattribute correction value, for example. The encoding context may beselected from a plurality of supported encoding contexts. For example, acontext store, such as context store 216 of an encoder, such as encoder202, as illustrated in FIG. 2A, may store a plurality of supportedencoding context for encoding attribute values or attribute correctionvalues for points of a point cloud. In some embodiments, an encodingcontext may be selected based on characteristics of a value to beencoded. For example, some encoding contexts may be optimized forencoding values with certain characteristics while other encodingcontexts may be optimized for encoding values with othercharacteristics.

In some embodiments, an encoding context may be selected based on aquantity or variety of symbols included in a value to be encoded. Forexample, values with fewer or less diverse symbols may be encoded usingarithmetic encoding techniques, while values with more symbols or morediverse symbols may be encoding using exponential Golomb encodingtechniques. In some embodiments, an encoding context may encode portionsof a value using more than one encoding technique. For example, in someembodiments, an encoding context may indicate that a portion of a valueis to be encoded using an arithmetic encoding technique and anotherportion of the value is to be encoded using a Golomb encoding technique.In some embodiments, an encoding context may indicate that a portion ofa value below a threshold is to be encoded using a first encodingtechnique, such as arithmetic encoding, whereas another portion of thevalue exceeding the threshold is to be encoded using another encodingtechnique, such as exponential Golomb encoding. In some embodiments, acontext store may store multiple encoding contexts, wherein eachencoding context is suited for values having particular characteristics.

At 506, a first value (or additional value) for the point may be encodedusing the encoding context selected at 504. At 508 it is determined ifthere are additional values for the point that are to be encoded. Ifthere are additional values for the point to be encoded, the additionalvalues may be encoded, at 506, using the same selected encodingtechnique that was selected at 504. For example, a point may have a“Red”, a “Green”, and a “Blue” color attribute. Because differencesbetween adjacent points in the R, G, B color space may be similar,attribute correction values for the Red attribute, Green attribute, andBlue attribute may be similar. Thus, in some embodiments, an encoder mayselect an encoding context for encoding attribute correction values fora first one of the color attributes, for example the Red attribute, andmay use the same encoding context for encoding attribute correctionvalues for the other color attributes, such as the Green attribute andthe Blue attribute.

At 510 encoded values, such as encoded assigned attribute values andencoded attribute correction values may be included in a compressedattribute information file. In some embodiments, the encoded values maybe included in the compressed attribute information file in accordancewith the evaluation order determined for the point cloud based on aminimum spanning tree. Thus a decoder may be able to determine whichencoded value goes with which attribute of which point based on theorder in which encoded values are included in a compressed attributeinformation file. Additionally, in some embodiments, data may beincluded in a compressed attribute information file indicatingrespective ones of the encoding contexts that were selected to encoderespective ones of the values for the points.

Exampled Processes for Encoding Spatial Information

FIGS. 6A-B illustrate an example process for compressing spatialinformation of a point cloud, according to some embodiments.

At 602, an encoder receives a point cloud. The point cloud may be acaptured point cloud from one or more sensors or may be a generatedpoint cloud, such as a point cloud generated by a graphics application.For example, 604 illustrates points of an un-compressed point cloud.

At 606, the encoder sub-samples the received point cloud to generate asub-sampled point cloud. The sub-sampled point cloud may include fewerpoints than the received point cloud. For example, the received pointcloud may include hundreds of points, thousands of points, or millionsof points and the sub-sampled point cloud may include tens of points,hundreds of points or thousands of points. For example, 608 illustratessub-sampled points of a point cloud received at 602, for example asub-sampling of the points of the point cloud in 604.

In some embodiments, the encoder may encode and decode the sub-sampledpoint cloud to generate a representative sub-sampled point cloud thedecoder will encounter when decoding the compressed point cloud. In someembodiments, the encoder and decoder may execute a lossycompression/decompression algorithm to generate the representativesub-sampled point cloud. In some embodiments, spatial information forpoints of a sub-sampled point cloud may be quantized as part ofgenerating a representative sub-sampled point cloud. In someembodiments, an encoder may utilize lossless compression techniques andencoding and decoding the sub-sampled point cloud may be omitted. Forexample, when using lossless compression techniques the originalsub-sampled point cloud may be representative of a sub-sampled pointcloud the decoder will encounter because in lossless compression datamay not be lost during compression and decompression.

At 610, the encoder identifies subdivision locations between points ofthe sub-sampled point cloud according to configuration parametersselected for compression of the point cloud or according to fixedconfiguration parameters. The configuration parameters used by theencoder that are not fixed configuration parameters are communicated toan encoder by including values for the configuration parameters in acompressed point cloud. Thus, a decoder may determine the samesubdivision locations as the encoder evaluated based on subdivisionconfiguration parameters included in the compressed point cloud. Forexample, 612 illustrates identified sub-division locations betweenneighboring points of a sub-sampled point cloud.

At 614, the encoder determines for respective ones of the subdivisionlocations whether a point is to be included or not included at thesubdivision location in a decompressed point cloud. Data indicating thisdetermination is encoded in the compressed point cloud. In someembodiments, the data indicating this determination may be a single bitthat if “true” means a point is to be included and if “false” means apoint is not to be included. Additionally, an encoder may determine thata point that is to be included in a decompressed point cloud is to berelocated relative to the subdivision location in the decompressed pointcloud. For example 616, shows some points that are to be relocatedrelative to a subdivision location. For such points, the encoder mayfurther encode data indicating how to relocate the point relative to thesubdivision location. In some embodiments, location correctioninformation may be quantized and entropy encoded. In some embodiments,the location correction information may comprise delta X, delta Y,and/or delta Z values indicating how the point is to be relocatedrelative to the subdivision location. In other embodiments, the locationcorrection information may comprise a single scalar value whichcorresponds to the normal component of the location correctioninformation computed as follows:

ΔN=([X _(A) ,Y _(A) ,Z _(A)]−[X,Y,Z])·[Normal Vector]

In the above equation, delta N is a scalar value indicating locationcorrection information that is the difference between the relocated oradjusted point location relative to the subdivision location (e.g.[X_(A),Y_(A),Z_(A)]) and the original subdivision location (e.g.[X,Y,Z]). The cross product of this vector difference and the normalvector at the subdivision location results in the scalar value delta N.Because a decoder can determine, the normal vector at the subdivisionlocation, and can determine the coordinates of the subdivision location,e.g. [X,Y,Z], the decoder can also determine the coordinates of theadjusted location, e.g. [X_(A),Y_(A),Z_(A)], by solving the aboveequation for the adjusted location, which represents a relocatedlocation for a point relative to the subdivision location. In someembodiments, the location correction information may be furtherdecomposed into a normal component and one or more additional tangentialcomponents. In such an embodiment, the normal component, e.g. delta N,and the tangential component(s) may be quantized and encoded forinclusion in a compressed point cloud.

In some embodiments, an encoder may determine whether one or moreadditional points (in addition to points included at subdivisionlocations or points included at locations relocated relative tosubdivision locations) are to be included in a decompressed point cloud.For example, if the original point cloud has an irregular surface orshape such that subdivision locations between points in the sub-sampledpoint cloud do not adequately represent the irregular surface or shape,the encoder may determine to include one or more additional points inaddition to points determined to be included at subdivision locations orrelocated relative to subdivision locations in the decompressed pointcloud. Additionally, an encoder may determine whether one or moreadditional points are to be included in a decompressed point cloud basedon system constraints, such as a target bitrate, a target compressionratio, a quality target metric, etc. In some embodiments, a bit budgetmay change due to changing conditions such as network conditions,processor load, etc. In such embodiments, an encoder may adjust aquantity of additional points that are encoded to be included in adecompressed point cloud based on a changing bit budget. In someembodiments, an encoder may include additional points such that a bitbudget is consumed without being exceeded. For example, when a bitbudget is higher, an encoder may include more additional points toconsume the bit budget (and enhance quality) and when the bit budget isless, the encoder may include fewer additional points such that the bitbudget is consumed but not exceeded.

In some embodiments, an encoder may further determine whether additionalsubdivision iterations are to be performed. If so, the points determinedto be included, relocated, or additionally included in a decompressedpoint cloud are taken into account and the process reverts to 610 toidentify new subdivision locations of an updated sub-sampled point cloudthat includes the points determined to be included, relocated, oradditionally included in the decompressed point cloud. In someembodiments, a number of subdivision iterations to be performed (N) maybe a fixed or configurable parameter of an encoder. In some embodiments,different subdivision iteration values may be assigned to differentportions of a point cloud. For example, an encoder may take into accounta point of view from which the point cloud is being viewed and mayperform more subdivision iterations on points of the point cloud in theforeground of the point cloud as viewed from the point of view and fewersubdivision iterations on points in a background of the point cloud asviewed from the point of view.

At 618, the spatial information for the sub-sampled points of the pointcloud are encoded. Additionally, subdivision location inclusion andrelocation data is encoded. Additionally, any configurable parametersselected by the encoder or provided to the encoder from a user areencoded. The compressed point cloud may then be sent to a receivingentity as a compressed point cloud file, multiple compressed point cloudfiles, or may be packetized and communicated via multiple packets to areceiving entity, such as a decoder or a storage device. In someembodiments, a compressed point cloud may comprise both compressedspatial information and compressed attribute information. In otherembodiments, compressed spatial information and compressed attributeinformation may be included is separate compressed point cloud files.

FIG. 7 illustrates another example process for compressing spatialinformation of a point cloud, according to some embodiments.

In some embodiments, other spatial information compression techniquesother than the sub-sampling and prediction spatial information techniquedescribed in FIGS. 6A-B may be used. For example, a spatial encoder,such as spatial encoder 204, or a spatial decoder, such as spatialdecoder 222, may utilize other spatial information compressiontechniques, such as a K-D tree spatial information compressiontechnique. For example, compressing spatial information at 406 of FIG. 4may be performed using a sub-sampling and prediction technique similarto what is described in FIGS. 6A-B, may be performed using a K-D treespatial information compression technique similar to what is describedin FIG. 7, or may be performed using another suitable spatialinformation compression technique.

In a K-D tree spatial information compression technique, a point cloudcomprising spatial information may be received at 702. In someembodiments, the spatial information may have been previously quantizedor may further be quantized after being received. For example 718illustrates a captured point cloud that may be received at 702. Forsimplicity, 718 illustrates a point cloud in two dimensions. However, insome embodiments, a received point cloud may include points in 3-Dspace.

At 704, a K-dimensional tree or K-D tree is built using the spatialinformation of the received point cloud. In some embodiments, a K-D treemay be built by dividing a space, such as a 1-D, 2-D, or 3-D space of apoint cloud in half in a predetermined order. For example, a 3-D spacecomprising points of a point cloud may initially be divided in half viaa plane intersecting one of the three axis, such as the X-axis. Asubsequent division may then divide the resulting space along anotherone of the three axis, such as the Y-axis. Another division may thendivide the resulting space along another one of the axis, such as theZ-axis. Each time a division is performed a number of points included ina child cell created by the division may be recorded. In someembodiments, only a number of points in one child cell of two childcells resulting from a division may be recorded. This is because anumber of points included in the other child cell can be determined bysubtracting the number of points in the recorded child cell from a totalnumber of points in a parent cell prior to the division.

A K-D tree may include a sequence of number of points included in cellsresulting from sequential divisions of a space comprising points of apoint cloud. In some embodiments, building a K-D tree may comprisecontinuing to subdivide a space until only a single point is included ineach lowest level child cell. A K-D tree may be communicated as asequence of number of points in sequential cells resulting fromsequential divisions. A decoder may be configured with informationindicating the subdivision sequence followed by an encoder. For example,an encoder may follow a pre-defined division sequence until only asingle point remains in each lowest level child cell. Because thedecoder may know the division sequence that was followed to build theK-D tree and the number of points that resulted from each subdivision(which is communicated to the decoder as compressed spatial information)the decoder may be able to reconstruct the point cloud.

For example, 720 illustrates a simplified example of K-D compression ina two-dimensional space. An initial space includes seven points. Thismay be considered a first parent cell and a K-D tree may be encoded witha number of points “7” as a first number of the K-D tree indicating thatthere are seven total points in the K-D tree. A next step may be todivide the space along the X-axis resulting in two child cells, a leftchild cell with three points and a right child cell with four points.The K-D tree may include the number of points in the left child cell,for example “3” as a next number of the K-D tree. Recall that the numberof points in the right child cell can be determined based on subtractingthe number of points in the left child cell from the number of points inthe parent cell. A further step may be to divide the space an additionaltime along the Y-axis such that each of the left and right child cellsare divided in half into lower level child cells. Again, a number ofpoints included in the left lower-level child cells may be included in aK-D tree, for example “0” and “1”. A next step may then be to divide thenon-zero lower-level child cells along the X-axis and record the numberof points in each of the lower-level left child cells in a K-D tree.This process may continue until only a single point remains in a lowestlevel child cell. A decoder may utilize a reverse process to recreate apoint cloud based on receiving a sequence of point totals for each leftchild cell of a K-D tree.

At 706, an encoding context for encoding a number of points for a firstcell of the K-D tree, for example the parent cell comprising sevenpoints, is selected. In some embodiments, a context store may storehundreds or thousands of encoding contexts. In some embodiments, cellscomprising more points than a highest number of points encoding contextmay be encoded using the highest number point encoding context. In someembodiments, an encoding context may include arithmetic encoding, Golombexponential encoding, or a combination of the two. In some embodiments,other encoding techniques may be used. In some embodiments, anarithmetic encoding context may include probabilities for particularsymbols, wherein different arithmetic encoding contexts includedifferent symbol probabilities.

At 708, the number of points for the first cell is encoded according theselected encoding context.

At 710, an encoding context for encoding a child cell is selected basedon a number of points included in a parent cell. The encoding contextfor the child cell may be selected in a similar manner as for the parentcell at 706.

At 712, the number of points included in the child cell is encodedaccording the selected encoding context, selected at 710. At 714, it isdetermined if there are additional lower-level child cells to encode inthe K-D tree. If so, the process reverts to 710. If not, at 716, theencoded number of points in the parent cell and the child cells areincluded in a compressed spatial information file, such as a compressedpoint cloud. The encoded values are ordered in the compressed spatialinformation file such that the decoder may reconstruct the point cloudbased on the number of points of each parent and child cell and theorder in which the number of points of the respective cells are includedin the compressed spatial information file.

In some embodiments, the number of points in each cell may be determinedand subsequently encoded as a group at 716. Or, in some embodiments, anumber of points in a cell may be encoded subsequent to being determinedwithout waiting for all child cell point totals to be determined.

Level of Detail Attribute Compression

In some circumstances, a number of bits needed to encode attributeinformation for a point cloud may make up a significant portion of a bitstream for the point cloud. For example, the attribute information maymake up a larger portion of the bit stream than is used to transmitcompressed spatial information for the point cloud.

In some embodiments, spatial information may be used to build ahierarchical Level of Detail (LOD) structure. The LOD structure may beused to compress attributes associated with a point cloud. The LODstructure may also enable advanced functionalities such asprogressive/view-dependent streaming and scalable rendering. Forexample, in some embodiments, compressed attribute information may besent (or decoded) for only a portion of the point cloud (e.g. a level ofdetail) without sending (or decoding) all of the attribute informationfor the whole point cloud.

FIG. 8 illustrates an example encoding process that generates ahierarchical LOD structure, according to some embodiments. For example,in some embodiments, an encoder such as encoder 202 may generatecompressed attribute information in a LOD structure using a similarprocess as shown in FIG. 8.

In some embodiments, geometry information (also referred to herein as“spatial information”) may be used to efficiently predict attributeinformation. For example, in FIG. 8 the compression of color informationis illustrated. However, a LOD structure may be applied to compressionof any type of attribute (e.g., reflectance, texture, modality, etc.)associated with points of a point cloud. Note that a pre-encoding stepwhich applies color space conversion or updates the data to make thedata better suited for compression may be performed depending on theattribute to be compressed.

In some embodiments, attribute information compression according to aLOD process proceeds as described below.

For example, let Geometry (G)={Point−P(0),P(1), . . . P(N−1)} bereconstructed point cloud positions generated by a spatial decoderincluded in an encoder (geometry decoder GD 802) after decoding acompressed geometry bit stream produced by a geometry encoder, alsoincluded in the encoder (geometry encoder GE 814), such as spatialencoder 204 (illustrated in FIG. 2A). For example, in some embodiments,an encoder such as encoder 202 (illustrated in FIG. 2A) may include botha geometry encoder, such as geometry encoder 814, and a geometrydecoder, such as geometry decoder 802. In some embodiments, a geometryencoder may be part of spatial encoder 214 and a geometry decoder may bepart of prediction/correction evaluator 206, both as illustrated in FIG.2A.

In some embodiments, the decompressed spatial information may describelocations of points in 3D space, such as X, Y, and Z coordinates of thepoints that make up mug 800. Note that spatial information may beavailable to both an encoder, such as encoder 202, and a decoder, suchas decoder 220. For example various techniques, such as K-D treecompression, octree compression, nearest neighbor prediction, etc., maybe used to compress and/or encode spatial information for mug 800 andthe spatial information may be sent to a decoder with, or in additionto, compressed attribute information for attributes of the points thatmake up a point cloud for mug, such as a point cloud 800.

In some embodiments, a deterministic re-ordering process may be appliedon both an encoder side (such as at encoder 202) and at a decoder side(such as at decoder 220) in order to organize points of a point cloud,such as the points that represent mug 800, into a set of Level ofDetails (LODs). For example, levels of detail may be generated by alevel of detail generator 804, which may be included in aprediction/correction evaluator of an encoder, such asprediction/correction evaluator 206 of encoder 202 as illustrated inFIG. 2A. In some embodiments, a level of detail generator 804 may be aseparate component of an encoder, such as encoder 202. For example,level of detail generator 804 may be a separate component of encoder202. Note that, in some embodiments, no additional information needs tobe included in the bit stream to generate such LOD structures, exceptfor the parameters of the LOD generation algorithm, For example,parameters that may be included in a bit stream as parameters of the LODgenerator algorithm may include:

-   -   i. The maximum number of LODs to be generated denoted by “N”        (e.g., N=6),    -   ii. The initial sampling distance “D0” (e.g., D0=64), and    -   iii. The sampling distance update factor “f” (e.g., ½).

In some embodiments, the parameters N, D0 and f, may be provided by auser, such as an engineer configuring a compression process. In someembodiments the parameters N, D0 and f, may be determined automaticallyby an encoder/and or decoder using an optimization procedure, forexample. These parameters may be fixed or adaptive.

In some embodiments, LOD generation may proceed as follows:

-   -   a. Points of geometry G (e.g. the points of the point cloud        organized according to the spatial information), such as points        of mug 800, are marked as non-visited and a set of visited        points V is set to be empty.    -   b. The LOD generation process may then proceed iteratively. At        each iteration j, the level of detail for that refinement level,        e.g. LOD(j), may be generated as follows:        -   1. The sampling distance for the current LOD, denoted D(j)            may be set as follows:            -   a. If j=0, then D(j)=D0.            -   b. If j>0 and j<N, then D(j)=D(j−1)*f.            -   c. if j=N, then D(j)=0.    -   2. The LOD generation process iterates over all the points of G.        -   a. At the point evaluation iteration i, a point P(i) is            evaluated,            -   i. if the point P(i) has been visited then it is ignored                and the algorithm jumps to the next iteration (i+1),                e.g. the next point P(i+1) is evaluated.            -   ii. Otherwise, the distance D(i,V), defined as the                minimum distance from P(i) over all the points of V, is                computed. Note that V is the list of points that have                already been visited. If V is empty, the distance D(i,V)                is set to 0, meaning that the distance from point P(i)                to the visited points is zero because there are not any                visited points in the set V. If the shortest distance                from point P(i) to any of the already visited point,                D(i,V), is strictly higher than a parameter D0, then the                point is ignored and the LoD generation jumps to the                iteration (i+1) and evaluates the next point P(i+1).                Otherwise, P(i) is marked as a visited point and the                point P(i) is added to the set of visited points V.        -   b. This process may be repeated until all the points of            geometry G are traversed.    -   3. The set of points added to V during the iteration j describes        the refinement level R(j).    -   4. The LOD(j) may be obtained by taking the union of all the        refinement levels R(0), R(1), . . . , R(j).

In some embodiments, the process described above, may be repeated untilall the LODs are generated or all the vertices have been visited.

In some embodiments, an encoder as described above may further include aquantization module (not shown) that quantizes geometry informationincluded in the “positions (x,y,z) being provided to the geometryencoder 814. Furthermore, in some embodiments, an encoder as describedabove may additionally include a module that removes duplicated pointssubsequent to quantization and before the geometry encoder 814.

In some embodiments, quantization may further be applied to compressedattribute information, such as attribute correction values and/or one ormore attribute value starting points. For example quantization isperformed at 810 to attribute correction values determined byinterpolation-based prediction module 808. Quantization techniques mayinclude uniform quantization, uniform quantization with a dead zone,non-uniform/non-linear quantization, trellis quantization, or othersuitable quantization techniques.

Example Level of Detail Hierarchy

FIG. 9A illustrates an example LOD, according to some embodiments. Notethat the LOD generation process may generate uniformly sampledapproximations (or levels of detail) of the original point cloud, thatget refined as more and more points are included. Such a feature makesit particularly adapted for progressive/view-dependent transmission andscalable rendering. For example, 904 may include more detail than 902,and 906 may include more detail than 904. Also, 908 may include moredetail than 902, 904, and 906.

The hierarchical LOD structure may be used to build an attributeprediction strategy. For example, in some embodiments the points may beencoded in the same order as they were visited during the LOD generationphase. Also, in some embodiments LODs may be generated concurrently withdetermining an attribute prediction strategy. Attributes of each pointmay be predicted by using the K-nearest neighbors that have beenpreviously encoded. In some embodiments, “K” is a parameter that may bedefined by the user or may be determined by using an optimizationstrategy. “K” may be static or adaptive. In the latter case where “K” isadaptive, extra information describing the parameter may be included inthe bit stream.

In some embodiments, different prediction strategies may be used. Forexample, one of the following interpolation strategies may be used, aswell as combinations of the following interpolation strategies, or anencoder/decoder may adaptively switch between the differentinterpolation strategies. The different interpolation strategies mayinclude interpolation strategies such as: inverse-distanceinterpolation, barycentric interpolation, natural neighborinterpolation, moving least squares interpolation, or other suitableinterpolation techniques. For example, interpolation based predictionmay be performed at an interpolation-based prediction module 808included in a prediction/correction value evaluator of an encoder, suchas prediction/correction value evaluator 206 of encoder 202. Also,interpolation based prediction may be performed at aninterpolation-based prediction module 808 included in a predictionevaluator of a decoder, such as prediction evaluator 224 of decoder 220.In some embodiments, a color space may also be converted, at color spaceconversion module 806, prior to performing interpolation basedprediction. In some embodiments, a color space conversion module 806 maybe included in an encoder, such as encoder 202. In some embodiments, adecoder may further included a module to convert a converted colorspace, back to an original color space.

In some embodiments, quantization may further be applied to attributeinformation. For example quantization may performed at quantizationmodule 810. In some embodiments, an encoder, such as encoder 202, mayfurther include a quantization module 810. Quantization techniquesemployed by a quantization module 810 may include uniform quantization,uniform quantization with a dead zone, non-uniform/non-linearquantization, trellis quantization, or other suitable quantizationtechniques.

In some embodiments, LOD attribute compression may be used to compressdynamic point clouds as follows:

-   -   a. Let FC be the current point cloud frame and RF be the        reference point cloud.    -   b. Let M be the motion field that deforms RF to take the shape        of FC.        -   i. M may be computed on the decoder side and in this case            information may not be encoded in the bit stream.        -   ii. M may be computed by the encoder and explicitly encoded            in the bit stream            -   1. M may be encoded by applying a hierarchical                compression technique as described herein to the motion                vectors associated with each point of RF (e.g. the                motion of RF may be considered as an extra attribute).            -   2. M may be encoded as a skeleton/skinning-based model                with associated local and global transforms.            -   3. M may be encoded as a motion field defined based on                an octree structure, which is adaptively refined to                adapt to motion field complexity.            -   4. M may be described by using any suitable animation                technique such as key-frame-based animations, morphing                techniques, free-form deformations, key-point-based                deformation, etc.        -   iii. Let RF′ be the point cloud obtained after applying the            motion field M to RF. The points of RF′ may be then used in            the attribute prediction strategy by considering not only            the “K” nearest neighbor points of FC but also those of RF′.

Furthermore, attribute correction values may be determined based oncomparing the interpolation-based prediction values determined atinterpolation-based prediction module 808 to original non-compressedattribute values. The attribute correction values may further bequantized at quantization module 810 and the quantitated attributecorrection values, encoded spatial information (output from the geometryencoder 802) and any configuration parameters used in the prediction maybe encoded at arithmetic encoding module 812. In some embodiments, thearithmetic encoding module, may use a context adaptive arithmeticencoding technique. The compressed point cloud may then be provided to adecoder, such as decoder 220, and the decoder may determine similarlevels of detail and perform interpolation based prediction to recreatethe original point cloud based on the quantized attribute correctionvalues, encoded spatial information (output from the geometry encoder802) and the configuration parameters used in the prediction at theencoder.

FIG. 9B illustrates an example compressed point cloud file comprisingLODs, according to some embodiments. Level of detail attributeinformation file 950 includes configuration information 952, point clouddata 954, and level of detail point attribute correction values 956. Insome embodiments, level of detail attribute information file 950 may becommunicated in parts via multiple packets. In some embodiments, not allof the sections shown in the level of detail attribute information file950 may be included in each packet transmitting compressed attributeinformation. In some embodiments, a level of detail attributeinformation file, such as level of detail attribute information file950, may be stored in a storage device, such as a server that implementsan encoder or decoder, or other computing device.

FIG. 10A illustrates a method of encoding attribute information of apoint cloud using an update operation, according to some embodiments.

At 1002, a point cloud is received by an encoder. The point cloud may becaptured, for example by one or more sensors, or may be generated, forexample in software.

At 1004, the points of the point cloud are ordered in an order based onthe respective positions of the points along a space-filling curve thatfills a 3D space of the point cloud. For example, a first pointencountered along a patch of the space-filling curve may be ordered as astarting point and a next point encountered along the space-fillingcurve may be ordered as a second point in the order according to thespace filling curve. The points of the point cloud are assigned indexvalues according to the order of the points along the space filingcurve. For example the starting point may be given an index value of “1”and the next point encountered may be given an index value of “2”, etc.

At 1006, one or more level of details are generated, as describedherein. For example, FIG. 10B further discusses how level of detail maybe determined based on index values for points of a point cloud orderedaccording to a space-filling curve. Note that in some embodiments, thespatial information used at 1004 to determine the order according to thespace-filling curve may have been encoded or compressed and de-coded ordecompressed to generate a representative decompressed point cloudgeometry that a decoder would encounter. This representativedecompressed point cloud geometry may then be used to generate LODstructures as further described in FIG. 10B.

At 1008, an interpolation based prediction is performed to predictattribute values for the attributes of the points of the point cloud. At1010, attribute correction values are determined based on comparing thepredicted attribute values to original attribute values. For example, insome embodiments, an interpolation based prediction may be performed foreach level of detail to determine predicted attribute values for pointsincluded in the respective levels of detail. These predicted attributevalues may then be compared to attribute values of the original pointcloud prior to compression to determine attribute correction values forthe points of the respective levels of detail. For example, aninterpolation based prediction process as described in FIG. 1B, FIGS.4-5, and FIG. 8 may be used to determine predicted attribute values forvarious levels of detail. In some embodiments, attribute correctionvalues may be determined for multiple levels of detail of a LODstructure. For example a first set of attribute correction values may bedetermined for points included in a first level of detail and additionalsets of attribute correction values may be determined for pointsincluded in other levels of detail.

At 1012, an update operation may optionally be applied that affects theattribute correction values determined at 1010. Performance of theupdate operation is discussed in more detail below in FIG. 12A-B.

In some embodiments, levels of detail may be determined using abottom-up approach, wherein a lowest level of detail comprising a sparsenumber of points is determined first and subsequent levels of detail addpoints to each preceding level of detail, such that each subsequentlevel of detail includes all the points of the preceding level of detailplus additional points that have been added for the subsequent level ofdetail to further refine the preceding level of detail.

In some embodiments, after determine points to include in a first levelof detail or a subsequent level of detail an encoder may determinepredicted attribute values and attribute correction values for the firstor subsequent level of detail while continuing to determine points toinclude in additional levels of detail. For example, an encoder mayconcurrently determine points to include in higher levels of detailwhile determining attribute correction values or updated attributecorrection values to encode for lower levels of detail.

In some embodiments, at 1008, instead of performing an absolute nearestneighbor search to determine nearest neighboring points to use in theinverse-distance based interpolations, the encoder may perform anapproximate nearest neighbor search. For example, the encoder maydetermine a sub-group of the points of the point cloud to be evaluatedas part of a nearest neighbor search based on index values of theneighboring points and an index value of a point for which nearestneighbors is being determined. For example, an encoder may considerpoints having index values with a second search range (SR2) from anindex value of the point being evaluated. This may simplify the nearestneighbor search while having minimal impact on the quality of attributeprediction.

In some embodiments, a first search range (SR1) used in determiningpoints to include in a level of detail and a second search range (SR2)used in a simplified nearest neighbor search for attribute valueprediction may be user specified parameters. Also, as discussed above, anumber of nearest neighbors to consider may be a user specifiedparameter. Also, in some embodiments, an initial inclusion distance (D0)and an inclusion distance ratio between layers (rho) may also be userdefined parameters. In some embodiments, a user may be an engineercustomizing the encoder/decoder parameters for use in a particularcompression application.

At 1014, attribute correction values, LOD parameters, encoded spatialinformation (output from the geometry encoder) and any configurationparameters used in the prediction are encoded, as described herein.

In some embodiments, the attribute information encoded at 1014 mayinclude attribute information for multiple or all levels of detail ofthe point cloud, or may include attribute information for a single levelof detail or fewer than all levels of detail of the point cloud. In someembodiments, level of detail attribute information may be sequentiallyencoded by an encoder. For example, an encoder may make available afirst level of detail before encoding attribute information for one ormore additional levels of detail.

In some embodiments, an encoder may further encode one or moreconfiguration parameters to be sent to a decoder, such as any of theconfiguration parameters shown in configuration information 952 ofcompressed attribute information file 950. For example, in someembodiments, an encoder may encode a number of levels of detail that areto be encoded for a point cloud. The encoder may also encode a samplingdistance update factor (e.g. inclusion distance ratio between layers),wherein the sampling distance is used to determine which points are tobe included in a given level of detail.

FIG. 10B illustrates an example process determining levels of detail,according to some embodiments.

At 1052, an encoder or decoder assigns index values to points of thepoint cloud based on respective positions of the points along thespace-filling curve. For example, the points may be assigned indexvalues according to a Morton order.

At 1054, the encoder or decoder selects a first or next level of detailto evaluate to determine which points of the point cloud will beincluded in the level of detail being evaluated. As mentioned above, thelevels of detail may be determined in a bottom-up manner from a leastpopulated level of detail to a most populated level of detail.

At 1056, the encoder or decoder selects a first or next point toevaluate. At 1058, the encoder or decoder identifies a set ofneighboring points having index values within a search range (SR1) of anindex value of the point being evaluated. At 1060, the encoder ordecoder determines respective distances between the point beingevaluated and each of the neighboring points in the search range (SR1).At 1062, the encoder or decoder includes in the current level of detailbeing evaluated neighboring points with respective distances to thepoint being evaluated that are less than an inclusion distance in thecurrent level of detail being evaluated. Note that points alreadyincluded in a lower level of detail may not be available to be selectedas nearest neighboring or points within the search range (SR1).

At 1064, the encoder or decoder determines whether there are additionalpoints to evaluate for the current level of detail being evaluated. Ifso, the process reverts to 1056 and a next point is evaluated. If not,at 1066, the encoder or decoder determines whether there are additionallevels of detail to evaluate. If not, at 1068 the level of detailrefinement process is completed and the determined levels of detail areused to predict attribute values for the respective points of therespective levels of detail. In some embodiments, attribute predictionfor a lower level of detail may be performed prior to completing thelevel of detail process for all higher levels of detail.

If there are additional levels of detail to evaluate, at 1070, theinclusion distance is increased to an inclusion distance for the nexthigher level of detail and the process reverts to 1054 to determinepoints to be included in the next level of detail. In some embodiments,a bottom-up approach as described in more detail below may be used.

In some embodiments, a lifting scheme as described below in regard toFIGS. 12A-12B may further implement a bottom-up approach to building thelevels of detail (LOD) as described in FIG. 10B and below. For example,instead of determining predicted values for points and then assigningthe points to different levels of detail, the predicted values may bedetermined while determining which points are to be included in whichlevel of detail. Also, in some embodiments, residual values (e.g.attribute correction values or updated attribute correction values) maybe determined by comparing the predicted values to the actual values ofthe original point could. This too may be performed while determiningwhich points are to be included in which levels of detail. Also, in someembodiments, an approximate nearest neighbor search may be used insteadof an exact nearest neighbor search to accelerate level of detailcreation and prediction calculations. In some embodiments, abinary/arithmetic encoder/decoder may be used to compress/decompressquantized computed wavelet coefficients.

As discussed above, a bottom-up approach may build levels of detail(LODs) and compute predicted attribute values simultaneously. In someembodiments, such an approach may proceed as follows:

-   -   Let (P_(i))_(i=1 . . . N) be the set of positions associated        with the point cloud points and let (M₃)_(i=1 . . . N) be the        Morton codes (or other order according to a space-filling curve)        associated with (P_(i))_(i=1 . . . N). Let D₀ and ρ be two        user-defined parameters specifying the initial sampling distance        and the distance ratio between LODs, respectively. A Morton code        (or other order according to a space-filling curve) may be used        to represent multi-dimensional data in one dimension, wherein a        “Z-Order function” is applied to the multidimensional data to        result in the one dimensional representation. Note that ρ>1    -   First the points are sorted according to their associated Morton        codes (or other order according to a space-filling curve) in an        ascending order. Let I be the array of point indexes ordered        according to this process.    -   The algorithm proceeds iteratively. At each iteration k, the        points belonging to the LOD k are extracted and their predictors        are built starting from k=0 until all the points are assigned to        an LOD.    -   The sampling distance D is initialized with D=D₀    -   For each iteration k, where k=0 . . . Number of LODs        -   Let L(k) be the set of indexes of the points belonging to            k-th LOD and O(k) the set of points belonging to LODs higher            than k. L(k) and O(k) are computed as follows.        -   First, O(k) and L(k) are initialized            -   if k=0, L(k)←{ }. Otherwise, L(k)←L(k−1)            -   O(k)←{ }        -   The point indexes stored in the array I are traversed in            order. Each time an index i is selected and its distance            (e.g., Euclidean or other distance) to the most recent SR1            points added to O(k) is computed. SR1 is a user-defined            parameter that controls the accuracy of the nearest neighbor            search. For instance, SR1 could be chosen as 8 or 16 or 64,            etc. The smaller the value of SR1 the lower the            computational complexity and the accuracy of the nearest            neighbor search. The parameter SR1 is included in the bit            stream. If any of the SR1 distances is lower than D, then i            is appended to the array L(k). Otherwise, i is appended to            the array O(k).            -   In some embodiments, the parameter SR1 could be changed                adaptively based on the LOD or/and the number of points                traversed. For example, similar parameters as discussed                above in regard to adaptive distance based interpolation                may also be used to adjust SR1.            -   In some embodiments, instead of computing an approximate                nearest neighbor, an exact nearest neighbor search                technique may be applied. For example, all points may be                considered for inclusion as nearest neighbors instead of                just points within a search range (SR1).            -   In some embodiments, the exact and approximate neighbor                search methods could be combined. In particular,                depending on the LOD and/or the number of points in I,                the method could switch between the exact and                approximate search method. Other criteria, may include                the point cloud density, the distance between the                current point and the previous one, or any other                criteria related to the point cloud distribution.        -   This process is iterated until all the indexes in I are            traversed.        -   At this stage, L(k) and O(k) are computed and will be used            in the next steps to build the predictors associated with            the points of L(k).

In some embodiments, a lifting scheme may be applied without determininga hierarchy of levels. In such embodiments, the technique may proceed asfollows:

-   -   Sort the input points according to the Morton codes associated        with their coordinates    -   Encode/decode point attributes according to the Morton order    -   For each point i, look for the h nearest neighbors (n₁, n₂, . .        . , n_(h)) already processed (n_(j)<i)    -   Compute the prediction weights as described above.    -   Apply the adaptive scheme described above in order to adjust the        prediction strategy.    -   Predict attributes and entropy encode them as described below.

Example Decoding Process for Bottom-Up LODs

FIG. 11 illustrates a method of decoding attribute information of apoint cloud, according to some embodiments.

At 1102, compressed attribute information for a point cloud is receivedat a decoder. Also, at 1104 spatial information for the point cloud isreceived at the decoder. In some embodiments, the spatial informationmay be compressed or encoded using various techniques, such as a K-Dtree, Octree, neighbor prediction, etc. and the decoder may decompressand/or decode the received spatial information at 1104.

At 1106, the decoder determines an order of the points of the pointcloud based on a space-filling curve. For example, the decoder mayrecreate a spatial representation of the point cloud based on thespatial information received at 1104 and determine Morton codes of thepoints of the point cloud. Also the decoder may determine which level ofdetail of a number of levels of detail to decompress/decode first ornext. The selected level of detail to decompress/decode may bedetermined based on a viewing mode of the point cloud. For example, apoint cloud being viewed in a preview mode may require fewer levels ofdetail to be determined than a point cloud being viewed in a full viewmode. Also, a location of a point cloud in a view being rendered may beused to determine a level of detail to decompress/decode. For example, apoint cloud may represent an object such as the coffee mug shown in FIG.8. If the coffee mug is in a foreground of a view being rendered morelevels of detail may be determined for the coffee mug. However, if thecoffee mug is in the background of a view being rendered, fewer levelsof detail may be determined for the coffee mug. In some embodiments, anumber of levels of detail to determine for a point cloud may bedetermined based on a data budget allocated for the point cloud.

At 1108 points included in the first level of detail (or next level ofdetail) being determined may be determined as described herein. For thepoints of the level of detail being evaluated, attribute values of thepoints may be predicted based on an inverse distance weightedinterpolation based on the k-nearest neighbors (also referred to hereinas “h”-nearest neighbors for an approximate nearest neighbor search) toeach point being evaluated, where k (or h) may be a fixed or adjustableparameter. Also, the nearest neighbor search may be an approximatenearest neighbor search that evaluates only points within a search range(SR1) of a particular point being evaluated based on an index value ofthe point being evaluated and index values of the neighboring points inthe order according to the space filling curve instead of evaluating allpoints of the point cloud to determine nearest neighbors.

At 1110, in some embodiments, an update operation may be performed onthe predicted attribute values as described in more detail in FIGS.12A-12B.

At 1112, attribute correction values included in the compressedattribute information for the point cloud may be decoded for the currentlevel of detail being evaluated and may be applied to correct theattribute values predicted at 1108 or the updated predicted attributevalues determined at 1110.

At 1114, the corrected attribute values determined at 1112 may beassigned as attributes to the points of the first level of detail (orthe current level of detail being evaluated). In some embodiments, theattribute values determined for subsequent levels of details may beassigned to points included in the subsequent levels of detail whileattribute values already determined for previous levels of detail areretained by the respective points of the previous level(s) of detail. Insome embodiments, new attribute values may be determined for sequentiallevels of detail.

In some embodiments, the spatial information received at 1104 mayinclude spatial information for multiple or all levels of detail of thepoint cloud, or may include spatial information for a single level ofdetail or fewer than all levels of detail of the point cloud. In someembodiments, level of detail attribute information may be sequentiallyreceived by a decoder. For example, a decoder may receive a first levelof detail and generate attribute values for points of the first level ofdetail before receiving attribute information for one or more additionallevels of detail.

At 1116 it is determined if there are additional levels of detail todecode. If so, the process returns to 1108 and is repeated for the nextlevel of detail to decode. If not the process is stopped at 1118, butmay resume at 1106 in response to input affecting the number of levelsof detail to determine, such as change in view of a point cloud or azoom operation being applied to a point cloud being viewed, as a fewexamples of an input affecting the levels of detail to be determined.

Lifting Schemes for Level of Detail Compression and Decompression

In some embodiments, lifting schemes may be applied to point clouds. Forexample, as described below, a lifting scheme may be applied toirregular points. This is in contrast to other types of lifting schemesthat may be applied to images having regular points in a plane. In alifting scheme, for points in a current level of detail nearest pointsin a lower level of detail are found. These nearest points in the lowerlevel of detail are used to predict attribute values for points in ahigher level of detail. Conceptually, a graph could be made showing howpoints in lower levels of detail are used to determine attribute valuesof points in higher levels of detail. In such a conceptual view, edgescould be assigned to the graph between levels of detail, wherein thereis an edge between a point in a higher level of detail and each point inthe lower level of detail that forms a basis for the prediction of theattribute of the point at the higher level of detail. As described inmore detail below, a weight could be assigned to each of these edgesindicating a relative influence. The weight may represent an influencean attribute value of the point in the lower level of detail has on theattribute value of the points in the higher level of detail. Also,multiple edges may make a path through the levels of detail and weightsmay be assigned to the paths. In some embodiments, the influence of apath may be defined by the sum of the weights of the edges of the path.For example, equation 1 discussed further below represents such aweighting of a path.

In a lifting scheme, attribute values for low influence points may behighly quantized and attribute values for high influence points may bequantized less. In some embodiments, a balance may be reached betweenquality of a reconstructed point cloud and efficiency, wherein morequantization increases compression efficiency and less quantizationincreases quality. In some embodiments, all paths may not be evaluated.For example, some paths with little influence may not be evaluated.Also, an update operator may smooth residual differences, e.g. predictedattribute values that are used to determine attribute correction values,in order to increase compression efficiency while taking into accountrelative influence or importance of points when smoothing the residualdifferences.

FIG. 12A illustrates a direct transformation that may be applied at anencoder to encode attribute information of a point could, according tosome embodiments.

In some embodiments, an encoder may utilize a direct transformation asillustrated in FIG. 12A in order to determine attribute correctionvalues that are encoded as part of a compressed point cloud. Forexample, in some embodiments a direct transformation, such asinterpolation based prediction, may be utilized to determine attributevalues as described in FIG. 10A at 1008 and to apply an update operationas described in FIG. 10A at 1012.

In some embodiments, a direct transform may receive attribute signalsfor attributes associated with points of a point cloud that is to becompressed. For example, the attributes may include color values, suchas RGB colors, or other attribute values of points in a point cloud thatis to be compressed. The geometry of the points of the point cloud to becompressed may also be known by the direct transform that receives theattribute signals. At 1202, the direct transform may include a splitoperator that splits the attribute signals 1210 for a first (or next)level of detail. For example, for a particular level of detail, such asLOD(N), comprising X number of points, a sub-sample of the attributes ofthe points, e.g. a sample comprising Y points, may comprise attributevalues for a smaller number of points than X. Said another way, thesplit operator may take as an input attributes associated with aparticular level of detail and generate a low resolution sample 1204 anda high resolution sample 1206. It should be noted that a LOD structuremay be partitioned into refinement levels, wherein subsequent levels ofrefinement include attributes for more points than underlying levels ofrefinement. A particular level of detail as described below is obtainedby taking the union of all lower level of detail refinements. Forexample, the level of detail j is obtained by taking the union of allrefinement levels R(0), R(1), . . . , R(j). It should also be noted, asdescribed above, that a compressed point cloud may have a total numberof levels of detail N, wherein R(0) is the least refinement level ofdetail and R(N) is the highest refinement level of detail for thecompressed point cloud.

At 1208, a prediction for the attribute values of the points notincluded in the low resolution sample 1204 is predicted based on thepoints included in the low resolution sample. For example, based on aninverse distance interpolation prediction technique or any of the otherprediction techniques described above. At 1212, a difference between thepredicted attribute values for the points left out of low resolutionsample 1204 is compared to the actual attribute values of the pointsleft out of the low resolution sample 1204. The comparison determinesdifferences, for respective points, between a predicted attribute valueand an actual attribute value. These differences (D(N)) are then encodedas attribute correction values for the attributes of the points includedin the particular level of detail that are not encoded in the lowresolution sample. For example, for the highest level of detail N, thedifferences D(N) may be used to adjust/correct attribute values includedin lower levels of detail. Because at the highest level of detail, theattribute correction values are not used to determine attribute valuesof other even higher levels of detail (because for the highest level ofdetail, N, there are not any higher levels of detail), an updateoperation to account for relative importance of these attributecorrection values may not be performed. As such, the differences D(N)may be used to encode attribute correction values for LOD(N).

In addition, the direct transform may be applied for subsequent lowerlevels of detail, such as LOD(N−1). However, before applying the directtransform for the subsequent level of detail, an update operation may beperformed in order to determine the relative importance of attributevalues for points of the lower level of detail on attribute values ofone or more upper levels of detail. For example, update operation 1214may determine relative importances of attribute values of attributes forpoints included in lower levels of detail on higher levels of detail,such as for attributes of points included in L(N). The update operatormay also smooth the attributes values to improve compression efficiencyof attribute correction values for subsequent levels of detail takinginto account the relative importance of the respective attribute values,wherein the smoothing operation is performed such that attribute valuesthat have a larger impact on subsequent levels of detail are modifiedless than points that have a lesser impact on subsequent levels ofdetail. Several approaches for performing the update operation aredescribed in more detail below. The updated lower resolution sample oflevel of detail L′(N) is then fed to another split operator and theprocess repeats for a subsequent level of detail, LOD(N−1). Note thatattribute signals for the lower level of detail, LOD(N−1) may also bereceived at the second (or subsequent) split operator.

FIG. 12B illustrates an inverse transformation that may be applied at adecoder to decode attribute information of a point cloud, according tosome embodiments.

In some embodiments, a decoder may utilize an inverse transformationprocess as shown in FIG. 11 to reconstruct a point cloud from acompressed point cloud. For example, in some embodiments, performingprediction as described in FIG. 11 at 1108, applying an update operatoras described in FIG. 11 at 1110, applying attribute correction values asdescribed in FIG. 11 at 1112 and assigning attributes to points in alevel of detail as described in FIG. 11 at 1114, may be performedaccording to an inverse transformation process as described in FIG. 12B.

In some embodiments, an inverse transformation process may receive anupdated low level resolution sample L′(0) for a lowest level of detailof a LOD structure. The inverse transformation process may also receiveattribute correction values for points not included in the updated lowresolution sample L′ (0). For example, for a particular LOD, L′(0) mayinclude a sub-sampling of the points included in the LOD and aprediction technique may be used to determine other points of the LOD,such as would be included in a high resolution sample of the LOD. Theattribute correction values may be received as indicated at 1206, e.g.D(0). At 1218 an update operation may be performed to account for thesmoothing of the attribute correction values performed at the encoder.For example, update operation 1218 may “undo” the update operation thatwas performed at 1214, wherein the update operation performed at 1214was performed to improve compression efficiency by smoothing theattribute values taking into account relative importance of theattribute values. The update operation may be applied to the updated lowresolution sample L′(0) to generate an “un-smoothed” or non-updated lowresolution sample, L(0). The low resolution sample L(0) may be used by aprediction technique at 1220 to determine attribute values of points notincluded in the low resolution sample. The predicted attribute valuesmay be corrected using the attribute correction values, D(0), todetermine attribute values for points of a high resolution sample of theLOD(0). The low resolution sample and the high resolution sample may becombined at merge operator 1222, and a new updated low resolution samplefor a next level of detail L′ (1) may be determined. A similar processmay be repeated for the next level of detail LOD(1) as was described forLOD(0). In some embodiments, an encoder as described in FIG. 12A and adecoder as described in FIG. 12B may repeat their respective processesfor N levels of detail of a point cloud.

In some embodiments, the bottom-up LODs as discussed above with regardto FIGS. 10A-10B may further be used in a lifting scheme as described inFIGS. 12A-12B.

-   -   a. More precisely, let R(k)=L(k)\L(k−1) (where \ is the        difference operator) be the set of points that need to be added        to LOD(k−1) to get LOD(k). For each point i in R(k), find the        h-nearest neighbors (h is user-defined parameters that controls        the maximum number of neighbors used for prediction) of i in        O(k) and compute the prediction weights (a_(j)(i))_(j=1 . . . h)        associated with i. The algorithm proceeds as follows.    -   b. Initialize a counter j=0    -   c. For each point i in R(k)        -   i. Let M_(i) be the Morton code associated with i and let            M_(j) be the Morton code associated with j-th element of the            array O(k)        -   ii. While (M_(i)≥M_(j) and j<SizeOf(O(k))), incrementing the            counter j by one (j←j+1)        -   iii. Compute the distances of M_(i) to the points associated            with the indexes of O(k) that are in the range [j−SR2,            j+SR2] of the array and keep track of the h-nearest            neighbors (n₁, n₂, . . . , n_(h)) and their associated            squared distances (d_(n) ₁ ²(i), d_(d) ₂ ²(i) . . . , d_(n)            _(h) ²(i)). SR2 is a user-defined parameter that controls            the accuracy of the nearest neighbor search. Possible values            for SR2 are 8, 16, 32, and 64. The smaller the value of SR2            the lower the computational complexity and the accuracy of            the nearest neighbor search. The parameter SR2 is included            in the bit stream. The computation of the prediction weights            used for attribute prediction may be the same as described            above.            -   a. The parameter SR2 could be changed adaptively based                on the LOD or/and the number of points traversed.            -   b. In some embodiments, instead of computing an                approximate nearest neighbor, an exact nearest neighbor                search technique may be used.            -   c. In some embodiments, the exact and approximate                neighbor search methods could be combined. In                particular, depending on the LOD and/or the number of                points in I, the method could switch between the exact                and approximate search method. Other criteria, may                include the point cloud density, the distance between                the current point and the previous one, or any other                criteria related to the point cloud distribution.            -   d. If the distance between the current point and the                last processed point is lower than a threshold, use the                neighbors of the last point as an initial guess and                search around them. The threshold could be adaptively                chosen based on similar criteria as those described                above. The threshold could be signaled in the bit stream                or known to both encoder and decoder.            -   e. The previous idea could be generalized to n=1,2,3,4 .                . . last points            -   f. Exclude points with a distance higher that a                user-defined threshold. The threshold could be                adaptively chosen based on similar criteria as those                described above. The threshold could be signaled in the                bitstream or known to both encoder and decoder.    -   d. I←O(k)    -   e. D←D×ρ    -   f. The approach described above could be used with any metric        (e.g., L2, L1, Lp) or any approximation of these metrics. For        example, in some embodiments distance comparisons may use a        Euclidean distance comparison approximation, such as a        Taxicab/Manhattan/L1 approximation, or an Octagonal        approximation.

More detailed example definitions of LODs and methods to determineupdate operations are described below.

In some embodiments, LODs are defined as follows:

LOD(0) = R(0) LOD(1) = LOD(0)UR(1) … LOD(j) = LOD(j − 1)UR(j) …LOD(N + 1) = LOD(N)UR(N) = entire  point  cloud

In some embodiments, let A be a set of attributes associated with apoint cloud. More precisely, let A(P) be the scalar/vector attributeassociated with the point P of the point cloud. An example of attributewould be color described by RGB values.

Let L(j) be the set of attributes associated with LOD(j) and H(j) thoseassociated with R(j). Based on the definition of level of detailsLOD(j), L(j) and H(j) verify the following properties:

-   -   L(N+1)=A and H(N+1)={ }    -   L(j)=L(j−1) U H(j)    -   L(j) and H(j) are disjointed.

In some embodiments, a split operator, such as split operator 1202,takes as input L(j+1) and generates two outputs: (1) the low resolutionsamples L(j) and (2) the high resolution samples H(j).

In some embodiments, a merge operator, such as merge operator 1222,takes as input L(j) and H(j) and produces L(j+1).

As described in more detail above, a prediction operator may be definedon top of an LOD structure. Let (P(i,j))_i be the set points of LOD(j)and (Q(i,j))_i those belonging to R(j) and let (A(P(i,j)))_i and(A(Q(i,j)))_i be the attribute values associated with LOD(j) and R(j),respectively.

In some embodiments, a prediction operator predicts the attribute valueA(Q(i,j)) by using the attribute values of its k nearest neighbors (orh-approximate nearest neighbors) in LOD(j−1), denoted ∇(Q(i,j)):

${{Pred}\left( {Q\left( {i,j} \right)} \right)} = {\sum\limits_{P \in {\nabla{({Q{({i,j})}})}}}{{\alpha\left( {P,{Q\left( {i,j} \right)}} \right)}{A(P)}}}$

where α(P,Q(i,j)) are the interpolation weights. For instance, aninverse distance weighted interpolation strategy may be exploited tocompute the interpolation weights.

The prediction residuals, e.g. attribute correction values, D(Q(i,j))are defined as follows:

D(Q(i,j))=A(Q(i,j))−Pred(Q(i,j))

Note that the prediction hierarchy could be described by an orientedgraph G defined as follows:

-   -   Every point Q in point cloud corresponds to a vertex V(Q) of        graph G.    -   Two vertices of the graph G, V(P) and V(Q), are connected by an        edge E(P,Q), if there exist i and j such that

Q=Q(i,j) and

P ∈ ∇(Q(i,j))

-   -   The edge E(Q,P), has weight α(P,Q(i,j)).

In such a prediction strategy as described above, points in lower levelsof detail are more influential since they are used more often forprediction.

Let w(P) be the influence weight associated with a point P. w(P) couldbe defined in various ways.

Approach 1

-   -   Two vertices V(P) and V(Q) of G are said to be connected if        there is a path x=(E(1), E(2), . . . , E(s)) of edges of G that        connects them. The weight w(x) of the path x is defined, as        follows:

${w(x)} = {\prod\limits_{s = 1}^{s}{\alpha\left( {E(s)} \right)}}$

-   -   Let X(P) be the set of paths having P as destination. w(P) is        defined as follows:

w(P)=1+Σ_(x∈X(P))(w(x))²   [EQ. 1]

-   -   The previous definition could be interpreted as follows. Suppose        that the attribute A(P) is modified by an amount ε, then all the        attributes associated with points connected to P are perturbed.        Sum of Squared Errors associated with such perturbation, denoted        SSE(P,ε) is given by:

SSE(P,ε)=w(P)ε²

Approach 2

-   -   Computing the influence weights as described previously may be        computationally complex, because all the paths need to be        evaluated. However, since the weights α(E(s)) are usually        normalized to be between 0 and 1, the weight w(x) of a path x        decays rapidly with the number of its edges. Therefore, long        paths could be ignored without significantly impacting the final        influence weight to be computed.    -   Based on the previous property, the definition in [EQ. 1] may be        modified to only consider paths with a limited length or to        discard paths with weights known to be lower that a user-defined        threshold. This threshold could be fixed and known at both the        encoder and decoder, or could be explicitly signaled at or        predefined for different stages of the encoding process, e.g.        once for every frame, LOD, or even after a certain number of        signaled points.

Approach 3

-   -   w(P) could be approximated by the following recursive procedure:        -   Set w(P)=1 for all points        -   Traverse the points according to the inverse of the order            defined by the LOD structure        -   For every point Q(i,j), update the weights of its neighbors            P ∈ ∇(Q(i,j)) as follows:

w(P)←w(P)+w(Q(i,j),j){α(P,Q(i,j))}^(γ)

-   -     where γ is a parameter usually set to 1 or 2.

Approach 4

-   -   w(P) could be approximated by the following recursive procedure:        -   Set w(P)=1 for all points        -   Traverse the points according to the inverse of the order            defined by the LOD structure        -   For every point Q(i,j), update the weights of its neighbors            P ∈ ∇(Q(i,j)) as follows:

w(P)←w(P)+w(Q(i,j),j)f{α(P,Q(i,j))}

-   -     where f(x) is some function with resulting values in the range        of [0, 1].

In some embodiments, an update operator, such as update operator 1214 or1218, uses the prediction residuals D(Q(i,j)) to update the attributevalues of LOD(j). The update operator could be defined in differentways, such as:

Approach 1

-   -   1. Let Δ(P) be the set of points Q(i,j) such that P ∈ ∇(Q(i,j)).    -   2. The update operation for P is defined as follows:

${Update}\mspace{11mu}{(P) = \frac{\Sigma_{Q \in {\Delta{(P)}}}\left\lbrack {\left\{ {\alpha\left( {P,Q} \right)} \right\}^{\gamma} \times {w(Q)} \times {D(Q)}} \right\rbrack}{\Sigma_{Q \in {\Delta{(P)}}}\left\lbrack {\left\{ {\alpha\left( {P,Q} \right)} \right\}^{\gamma} \times {w(Q)}} \right\rbrack}}$

-   -   where γ is a parameter usually set to 1 or 2.

Approach 2

-   -   1. Let Δ(P) be the set of points Q(i,j) such that P ∈ ∇(Q(i,j)).    -   2. The update operation for P is defined as follows:

${{Update}\mspace{11mu}(P)} = \frac{\Sigma_{Q \in {\Delta{(P)}}}\left\lbrack {g\left\{ {\alpha\left( {P,Q} \right)} \right\} \times {w(Q)}{D(Q)}} \right\rbrack}{\Sigma_{Q \in {\Delta{(P)}}}\left\lbrack {g\left\{ {\alpha\left( {P,Q} \right)} \right\} \times {w(Q)}} \right\rbrack}$

-   -   where g(x) is some function with resulting values in the range        of [0, 1].

Approach 3

-   -   Compute Update(P) iteratively as follows:        -   1. Initially set Update(P)=0        -   2. Traverse the points according to the inverse of the order            defined by the LOD structure        -   3. For every point Q(i,j), compute the local updates (u(1),            u(2), . . . , u(k)) associated with its neighbors            ∇(Q(i,j))={P(1), P(2), . . . , P(k)} as the solution to the            following minimization problem:

(u(1),u(2), . . ,u(k))=arg min{Σr=1 ^(k)(u(r))²+(D(Q(i,j))−Σ_(r=1)^(k)α(P(r),Q(i,j))u(k))²}

-   -     4. Update Update(P(r)):

Update(P(r))←Update(P(r))+u(r)

Approach 4

-   -   Compute Update(P) iteratively as follows:        -   1. Initially set Update(P)=0        -   2. Traverse the points according to the inverse of the order            defined by the LOD structure        -   3. For every point Q(i,j), compute the local updates (u(1),            u(2), . . . , u(k)) associated with its neighbors            ∇(Q(i,j))={P(1), P(2), . . . , P(k)} as the solution to the            following minimization problem:

(u(1),u(2), . . . ,u(k))=arg min{h(u(1), . . . ,u(k),D(Q(i,j)))}

-   -      Where h can be any function.        -   4. Update Update(P(r)):

Update(P(r))←Update(P(r))+u(r)

In some embodiments, when leveraging a lifting scheme as describedabove, a quantization step may be applied to computed waveletcoefficients. Such a process may introduce noise and a quality of areconstructed point cloud may depend on the quantization step chosen.Furthermore, as discussed above, perturbing the attributes of points inlower LODs may have more influence on the quality of the reconstructedpoint cloud than perturbing attributes of points in higher LODs.

In some embodiments, the influence weights computed as described abovemay further be leveraged during the transform process in order to guidethe quantization process. For example, the coefficients associated witha point P may be multiplied with a factor of {w(P)}^(β), where β is aparameter usually set to β=0.5. An inverse scaling process by the samefactor is applied after inverse quantization on the decoder side.

In some embodiments, the values of the β parameters could be fixed forthe entire point cloud and known at both the encoder and decoder, orcould be explicitly signaled at or predefined for different stages ofthe encoding process, e.g. once for every point cloud frame, LOD, oreven after a certain number of signaled points.

In some embodiments, a hardware-friendly implementation of the liftingscheme described above may leverage a fixed-point representation of theweights and lookup tables for the non-linear operations.

In some embodiments, a lifting scheme as described herein may beleveraged for other applications in addition to compression, such asde-noising/filtering, watermarking, segmentation/detection, as well asvarious other applications.

In some embodiments, a decoder may employ a complimentary process asdescribed above to decode a compressed point cloud compressed using anoctree compression technique and binary arithmetic encoder as describedabove.

Binary Arithmetic Coding of Quantized Lifting Coefficients UsingKey-Word Mapping

In some embodiments, lifting scheme coefficients may be non-binaryvalues. In some embodiments, an arithmetic encoder may be included as acomponent of encoder 202 illustrated in FIG. 2A and may use a binaryarithmetic codec to encode lifting scheme coefficients (e.g. updatedattribute correction values) using key-word mapping and a look-up table.

FIG. 13 illustrates a key-word mapping process using a look-up tablethat may be used to compress updated attribute correction values,according to some embodiments.

At 1302, the encoder selects a lifting scheme coefficient (e.g. updatedattribute correction value) to encode and at 1304 the encoder applies aquantization to the lifting scheme coefficient (e.g. updated attributecorrection value).

At 1306, the encoder determines whether the lifting scheme coefficient(e.g. updated attribute correction value) has a value of zero or not. Ifthe value is zero, at 1308 the encoder encodes a zero value. If not, theencoder determines, at 1310, whether the lifting scheme coefficient(e.g. updated attribute correction value) has a value equal to orgreater than a highest N-bit value supported by a look-up table (e.g.255) (e.g. a greatest alphabet value). If the value is less than themaximum value, the encoder encodes, at 1312, an “M” bit code wordcorresponding to an “N” bit value for the lifting scheme coefficient(e.g. updated attribute correction value). The “M” bit code word may bean entry in the look-up table corresponding to the “N” bit value,wherein the “M” bit code word includes fewer bits than the “N” bitvalue. A decoder may maintain a similar look-up table and may be able todetermine the “N” bit value by looking up the “M” bit code word in thelook-up table maintained by the decoder.

If the “N” bit value is greater than a maximum “N” bit value included inthe look-up table, the encoder may, at 1314, encode an “M” bit code wordcorresponding to the greatest “N” bit value included in the look-uptable and additionally encode, at 1316, a difference between thegreatest “N” bit value in the look-up table and the actual value for thelifting scheme coefficient (e.g. updated attribute correction value). Insome embodiments, the encoder may encode the “M” bit word using a binaryarithmetic encoding component and may encode the difference usinganother encoding component, such as an exponential Golomb encodingcompoenent.

For example, in more detail, the technique may proceed as follows:

-   -   Mono-dimensional attribute        -   Let C be the quantized coefficient to be encoded. First C is            mapped to a positive number using a function that maps            positive numbers to even numbers and negative numbers to odd            numbers.        -   Let M(C) be the mapped value.        -   A binary value is then encoded to indicate whether C is 0 or            not        -   If C is not zero, then two cases are distinguished            -   If M(C) is higher or equal than alphabetSize (e.g.the                number of symbols supported by the binary arithmetic                encoding technique), then the value alphabetSize is                encoded by using the method described above. The                difference between M(C) and alphabetSize is encoded by                using an exponential Golomb coding            -   Otherwise, the value of M(C) is encoded using the method                described above.    -   Three-dimensional signal        -   Let C1, C2, C3 be the quantized coefficients to be encoded.            Let K1, and K2 be two indexes for the contexts to be used to            encode the quantized coefficients, C1, C2, and C3.        -   First C1, C2 and C3 are mapped to a positive number as            described above. Let M(C1), M(C2) and M(C3) be the mapped            values of C1, C2, and C3.        -   M(C1) is encoded.        -   M(C2) is encoded while choosing different contexts (i.e.,            binary arithmetic contexts and the binarization context)            based on the condition of whether C1 is zero or not.        -   M(C3) is encoded while choosing different contexts based on            the conditions C1 is zero or not and C2 is zero or not. If            C1 is zero it is known that the value is at least 16. If the            condition C1 is zero use the binary context K1, if the value            is not zero, decrement the value by 1 (it is known that the            value is at least one or more), then check the value is            below the alphabet size, if so encode the value directly.            Otherwise, encode maximum possible value for the alphabet            size. The difference between the maximum possible value for            the alphabet size and the value of M(C3) will then be            encoded using exponential Golomb encoding.    -   Multi-dimensional signal        -   The same approach described above could be generalized to a            d-dimensional signal. Here, the contexts to encode the k-th            coefficient depending on the values of the previous            coefficients (e.g., last 0, 1, 2, 3, . . . , k−1            coefficients).        -   The number of previous coefficients to consider could be            adaptively chosen depending on any of the criteria described            in the previous section for the selection of SR1 and SR2.

Alternative Low-Complexity Level of Detail Generation Procedure

As discussed above, a level of detail (LOD) structure partitions thepoint cloud into non-overlapping subsets of points referred to asrefinement levels, e.g. (R_1)_(l=0 . . . L−1). In some embodiments inwhich a distance-based approach is used to determine level of detailrefinement levels (such as those discussed above), the refinement levelsare determined according to a set of Euclidian distances (d_1)_(l=0 . .. L−1) specified by the user, in a way, that the entire point cloud isrepresented by the union of all the refinements levels. The level ofdetail 1,

LOD

_1, is obtained by taking the union of the refinement levels R_0, R_1, .. . , R 1 as follows:

-   -   LOD_0=R_0    -   LOD_1=LOD_0 ∪ R_1 . . .    -   LOD_1=LOD_(l−1) ∪ R_1 . . .    -   LOD_(L−1)=LOD_(L−2) ∪ R_(L) represents entire point cloud

Points in each refinement level R_1 are extracted in such a way that theEuclidian distances between the points in that particular LOD aregreater than or equal to a user defined threshold D. As thelevel-of-detail j increases, D decreases and more points are includedin-between the points in the lower LOD, therefore increasing the pointcloud reconstruction detail. Attributes of a point in R_1 are thenpredicted from k nearest-neighbor points in LOD_(l−1) (or theh-approximate nearest neighboring points). Finally, the predictionresidue (e.g. the attribute correction values), i.e. the differencebetween actual and predicted values of attributes, is encoded using anentropy encoder, e.g. an arithmetic encoder.

The distance-based LOD generation process tries to guarantee a uniformsampling throughout the different LODs (see FIG. 9A). Such a strategyoffers efficient prediction results for smooth attribute signals definedover uniformly or near uniformly sampled point clouds.

LOD Generation Using a Space Filling Curve

In some embodiments, a low-complexity LOD generation process thatutilizes a space filling curve to order points and determine refinementlevels may be used. The spatial information may be encoded using any ofthe techniques described above for encoding spatial information, such asK-D trees, octree encoding, sub-sampling and inter-point prediction,etc. In this way both the encoder and the decoder may know the spatiallocations of the points of the point cloud. However, instead ofdetermining which points are to be included in respective refinementlevels based on distances between the points as is described above, thepoints to be included in respective levels of details may be determinedby ordering the points according to their location along a space fillingcurve. For example, the points may be organized according to theirMorton codes. Alternatively, other space filling curves could be used.For example, techniques to map positions (e.g., in X, Y, Z coordinateform) to a space filling curve such as a Morton-order (or Z-order),Hillbert curve, Peano curve, and so on may be used. In this way all ofthe points of the point cloud that are encoded and decoded using thespatial information may be organized into an index in the same order onthe encoder and the decoder. In order to determine various refinementlevels, sampling rates for the ordered index of the points may bedefined. For example, to divide a point cloud into four levels ofdetail, an index that maps a Morton value to a corresponding point maybe sampled, for example at a rate of four, where every fourth indexedpoint is included in the lowest level refinement. For each additionallevel of refinement remaining points in the index that have not yet beensampled may be sampled, for example every third index point, etc. untilall of the points are sampled for a highest level of detail. Forexample, a low-complexity LOD generation process that utilizes a spacefilling curve to order points and determine refinement levels, mayproceed as follows:

-   -   First, the points (for which spatial information is already        known) may be ordered according to a space filling curve. For        example, the points may be ordered according to their Morton        codes, as an example.    -   Then, Let I_(L−1) be the set of ordered indexes and LOD_(L−1)        the associated LOD that represents the entire point cloud.    -   Next, define a set of sampling rates denoted        (k_(l))_(l=0 . . . L−1), where k_(l) is an integer describing        the sampling rate for the LOD l.        -   k_(l) can be automatically determined based on the            characteristics of the signal and/or the point cloud            distribution, previous statistics, or could be fixed.        -   k_(l) can be provided as user-defined parameter (e.g., 4).        -   The sampling rate_k_(l) could be further updated within an            LOD in order to better adapt to the point cloud            distribution. More precisely, the encoder may explicitly            encode in the bit stream for a predefined group of points            (e.g., each consecutive H=1024 points) different values or            updates to be applied to the latest available k_(l) value.    -   Next, the ordered array of indexes associated with LOD l=L−2,        L−3, . . . , 0, denoted as I_(l), is computed by subsampling        I_(l+1), while keeping one index out of every k_(l) indexes.    -   In some embodiments, different subsampling rates may be defined        per attribute (e.g., color, reflectance) and per channel (e.g.,        Y and U/V), etc.

Combined Ordering/Sampling LOD Method

In some embodiments, the low-complexity LOD generation process thatutilizes a space filling curve to order points and determine refinementlevels may be combined with a distance-based prediction method asdescribed in earlier sections above. For example, for portions of apoint cloud with smooth attribute signals that are regularly sampled, adistance-based attribute prediction strategy may be used. However, forportions of the point cloud that include non-smooth attribute signalsthat irregularly sampled, a low-complexity LOD generation process thatutilizes a space filling curve to order points and determine refinementlevels may be used. In some embodiments, switching between thedistance-based LOD generation process and the low-complexity LODgeneration process using a space filling curve may be operated at:

-   -   Group of points level,    -   LOD level,    -   Slice level, and/or    -   Frame level.

Adaptive Scanning Mode

In some embodiments, prediction between levels of detail may also beused to determine predicted attribute values for the various levels ofdetail. As discussed earlier, attribute correction values may also beencoded for points, wherein the attribute correction values represent adifference between a predicted value and an original or pre-compressionvalue for the attribute.

In some embodiments, instead of using a single prediction order, asdescribed above for the distance-based LOD generation process, aprediction mode in a low-complexity complexity LOD generation processusing a space filling curve may allow for improved coding efficiency byselecting the prediction direction that gives an improvedrate-distortion performance. For example, in one case LODO may use theMorton order and in another case may use the inverse Morton order. Inanother case the traversal of the points may start from the center orany point explicitly signaled by the encoder. In another case, the pointorder obtained after geometry decoding may be used. The scanning ordercould also be explicitly encoded in the bit stream or agreed betweenencoder and decoder. For example, the encoder may explicitly signal tothe decoder that a point should be skipped and processed at a latertime. Signaling of this mode and its associate parameters could be doneat the sequence/frame/tile/slice/LOD/group of points level.

In some embodiments, an adaptive scanning mode may be used in othercodecs where higher LODs also permit prediction of their samples fromcurrent LOD samples. This may of course impact the decoding process ofthat LOD (e.g. limit its parallelization capability). Parallelization,however, could still be achieved by defining “independent” decodinggroups within an LOD. Such groups may allow parallel decoding by notpermitting prediction across them. However, prediction using the lowerlevel LOD as well as decoded samples within the current LOD group may bepermitted.

In some embodiments, an encoder may select the appropriate adaptivescanning mode by utilizing rate distortion optimization (RDO)strategies. In some embodiments, an encoder may further take intoaccount various additional criteria, such as computational complexity,battery life, memory requirement, latency, pre-analysis, collectedstatistics of past frames (history), user feedback, etc.

Adaptive Scanning Offset Mode

In some embodiments, another mode that may be applied in alow-complexity complexity LOD generation process using a space fillingcurve may be an “alternate sampling phase/offset” mode. For example, anencoder may signal to the decoder the sampling offset (e.g., a samplingoffset 1 may provide better rate distortion (RD) performance than offset0) that is used for selecting which points should be sampled from thecurrent LOD when generating the next LOD level. For example, instead ofbeginning the sampling of the ordered Morton codes with the first Mortoncode for the first point, the sampling may begin at an offset value,e.g. the second, third, etc. Morton code. This could have an impact inperformance since this could alter the coding and prediction process foreach LOD. Signaling of the sampling offset could be done at thesequence/frame/tile/slice/LOD/group of points level.

Attribute Interleaving Mode

In some embodiments, attributes/attribute channels of the point cloudcould be interleaved at different levels when predicting andcoding/decoding the residual data (e.g. attribute correction values) ateach LOD level. In particular, interleaving could be done at:

-   -   Point level,    -   Group of points level,    -   LOD level,    -   Slice level, and/or    -   Frame level.

In some embodiments, interleaving may be done at the attribute channellevel. For example, color channels may be interleaved together, whileother attributes are only interleaved at the LOD level. However, in someembodiments, different combinations could be used for different types ofattribute data. The interleaving method could be fixed and known betweenencoder and decoder, but could also be adaptive and could be signaled atdifferent levels of the bit stream. The decision for the method usedcould be based on rate distortion (RD) criteria, pre-analysis, pastencoding statistics, encoding/decoding complexity, or some othercriteria that a user or system has determined.

Inter Attribute/Cross-Component Prediction

In some embodiments, different interleaving methods may permit alsointer-attribute/attribute channel prediction, which may result in morecoding benefits. For instance, for YCbCr data, the Cb and Cr colorcomponents could be predicted through their luma component. Such aprediction mode could be selected at various levels, such as:

-   -   Point level,    -   Group of points level,    -   LOD level,    -   Slice level, and/or    -   Frame level.

In some embodiments, different prediction methods may be used. Forexample, the prediction could use a linear or non-linear predictionmodel, where the parameters of the model (e.g., scale and offset in amodel of Chroma=a*Y+b) are also signaled to the decoder. Such parameterscould be estimated in the encoder using different methods, e.g. using aleast squares method. An alternative mode would be to combine theLuma-based prediction with the value generated through the conventionalprediction method. For instance, we the weighted average of theLuma-based and the distance-based predictors could be considered. Theweighting parameters could be explicitly encoded in the bit stream orcould be implicitly determined by the decoder. An encoder coulddetermine such parameters using, for example, rate distortion (RD) basedcriteria, or other methods that may take in account pre-analysis andpast statistics of the encoding process.

In some embodiments, the prediction strategy for the current pointattribute could also be adapted based on already encoded/decodedattribute/attribute channel values for the same point. For instance, ifthe Luma (or other X attribute) value is known for a given point,neighbors could be excluded from the prediction process for the givenpoint, where the neighbors have Luma (or X attribute) values that varygreatly from the Luma (or X attribute) values for the given point. Forexample, based on the variance between Luma (or X attribute) values,chroma components or other attributes of the neighboring points may beexcluded from prediction for the given point. In some embodiments,prediction may be done only with points that have similarcharacteristics, e.g. satisfy multiple attribute thresholds and not onlydistance ones. The encoder may explicitly signal which alreadyencoded/decoded attributes or attribute channels should be used for theprediction adaptation. Similarity could be determined based on athreshold T or a set of thresholds, assuming multiple attributes areconsidered for the prediction selection process, T could be signaled inthe bit stream. Such threshold/threshold sets could be signaled atdifferent levels of the coding process, e.g. groups of points, LOD,slices, tiles, frames, or sequence.

Prediction Adaptation

In some embodiments, the prediction adaptation process could alsoleverage various statistics related to the point cloud geometry (e.g.spatial information). For instance, it could include only points withthe same x and/or y and/or z value and exclude others (not only based ondistance but also based on other geometric criteria such as angles.Prediction could also include points that are limited in distance acrossone or more dimensions. For example, points that are within an overalldistance D may be included, while, however also these points are alsowithin a distance along the coordinate axes such as an X-distance (dx),a Y-distance (dy), or a Z-distance (dz) from the current point. Suchprediction mode could be selected/signaled at various levels.

Dependent LOD Encoder Optimization

In some embodiments, an encoder could select the encoding parameters forthe current LOD by considering not only its own distortion/codingperformance but also the distortion introduced while predicting the nextLOD. Such distortion could be computed for all attributes or a subset(e.g., the luma component only) of attributes. Such subset could bepredetermined by the user or some other means, e.g. by analyzing thedata and determining, which attribute is most active/has the mostenergy. For example, which attribute has the most impact on predictionof other attributes and prediction across multiple refinement levels orLODs. The distortion evaluated for the next LOD could be Mean SquareError (MSE)-based or could consider some other distortion criteria.Subsampling of the points in the next LOD could also be considered toreduce complexity. For example, only half the impacted samples in thatLOD could be considered in this computation. Sampling could be random,fixed based on some defined sampling process, or could also be based onthe characteristics of the signals, e.g. the LOD could be analyzed andthe most “important” points in the LOD could be considered. Importancecould be determined for example based on the magnitude of the attribute.

Fixed-Point Number Representation Implementation

In some embodiments, attribute prediction, determination of attributecorrection values, lifting schemes for level of detail compression ordecompression, and/or quantization can be performed using fixed-pointnumber representations instead of floating point number representations.A fixed point number representation may be more hardware friendly (e.g.execute more efficiently on the hardware) and may avoid floating pointrounding issues and/or dependencies between software and hardwareplatforms that may slow down computations performed using fixed pointnumber representations.

In some embodiments, a fixed point number may be represented using abinary fixed point representation with n digits (e.g. n=16) after theradix point to represent rational numbers.

In some embodiments, a fixed-point implementation may avoid divisionoperations. For example, division may be a computationally expensiveoperation. In some embodiments, in order to avoid division at least someof the following techniques may be used: look-up tables, multi-regionlook up tables, and/or multi-region look up tables combined withinterpolation.

For example, in some embodiments, a division of the integers a and b,may be computed as an approximated version of the ratio a/b. Thisapproach may avoid having to perform an explicit division operation,which may be costly from a computing perspective. Determining theapproximated version value for the ratio a/b may be determined usingonly multiplication, addition, and/or shifting operations, combined witha look-up table (LUT) of size 2^(m) (e.g., m=8). For example, anapproximation of the integer b, denoted {circumflex over (b)} iscomputed as follows:

{circumflex over (b)}=2^(s)b₀

Where the s and b₀ are two integers defined as follows

-   -   s is the smallest integer that verifies b×2^(−s)<2^(m)    -   b₀=round(b×2^(−s))

Next, a look up table (LUT_of dimension 2^(m) is computed offline asfollows:

${{{LUT}(i)} = {{round}{\;\;}\left( \frac{2^{n}}{i} \right)}},{i = \left\{ {1,2,\ldots\mspace{14mu},\ 2^{m}} \right\}}$

In the look-up table, LUT(i) gives the fixed-point representation of theinverse of an integer i={1, 2, . . . , 2^(m)}. The fixed-pointrepresentation of the approximation {circumflex over (b)}⁻¹ of theinverse of any integer b is obtained as follows:

{circumflex over (b)}⁻¹=LUT(b₀)2^(−s)

Also, the fixed-point representation of the approximation of

$\left( \frac{a}{b} \right),$

denoted

,

is given by:

$\hat{\left( \frac{a}{b} \right)} = {\left( {{a \times LU{T\ \left( b_{0} \right)}} + 2^{s - 1}} \right) \div 2^{s}}$

-   -   where ÷ is the integer division operator.

For example, let n=16, m=8, b=10703 and a=8009

2^(m) = 256 2^(n) = 65536s = 6  because  10703 × 2⁻⁶ = 167.234375 < 2^(m)b₀ = round  (167.234375) = 167 b̂ = 2^(s)b₀ = 64 × 167 = 10688${LU{T\left( b_{0} \right)}} = {{r\;{{ound}\left( \frac{65536}{167} \right)}} = 392}$ = (8009 × 392 + 32) ÷ 64 = 40955${{Error}\mspace{14mu}{abs}\mspace{14mu}\left( {\frac{8009}{10703} - \frac{40955}{65536}} \right)} = {0.00022502686390868387}$

In some embodiments, a fixed-point implementation may be used todetermine prediction weights for determining predicted attribute values.For example, fixed point implementations may be used in predictioncalculations, such as in 158 of FIG. 1B, 422 of FIG. 4, calculationsperformed in regard to FIG. 5, prediction calculations such as in 1008of FIG. 10A and/or 1108 of FIG. 11. An example fixed pointimplementation used in attribute prediction calculations is shown below,wherein a prediction operator predicts the attribute value A(P) of apoint P by using the attribute values of its k nearest neighbors ∇(P):

${{Pred}(P)} = {\sum\limits_{Q \in {\nabla{(P)}}}{{\alpha\left( {P,Q} \right)}{A(P)}}}$

The prediction weights α(P,Q) are defined as follows:

${\alpha\left( {P,\ Q} \right)} = \frac{\frac{1}{{{P - Q}}^{2}}}{\Sigma_{{Q\; 1} \in {\nabla{(P)}}}\frac{1}{{{P - {Q\; 1}}}^{2}}}$

-   -   Where ∥P−Q∥ is the Euclidian distance between the two points P        and Q. In some embodiments, other distances, other than a        Euclidian distance may be used.

The prediction weights have the following properties:

-   -   i. α(P,Q) ∈ [0,1]    -   ii. Σ_(Q∈∇(P)) α(P,Q)=¹

Since the points coordinates are quantized and represented by integers,the distances ∥P−Q∥² have also integer values. Computing the α(P,Q)requires a division, which is avoided by computing an approximatedversion denoted α(P,Q). This is done by leveraging the algorithmdescribed above in regard to fixed point division. For examplerespective cases for determining of a weighted prediction value a pointwith one, two, and three “K” nearest neighbors, are described below.

-   -   i. Case of one neighbor Q1        -   b. The solution is trivial α(P,Q1)=2^(n)    -   i. Case of two neighbors Q1 and Q2

c.  Let  d 1 = P − Q 1²  and  d 2 = P − Q 2²${d.\mspace{14mu}{\alpha\left( {P,{Q\; 1}} \right)}} = {\frac{\frac{1}{d\; 1}}{\frac{1}{d\; 1} + \frac{1}{d\; 2}} = \frac{d\; 2}{{d\; 1} + {d\; 2}}}$

-   -     e. Approximate α(P,Q1) by applying the division approximation        algorithm with a=d2 and b=d1+d2        -   f. Approximate α(P,Q2) with (2^(n)−α(P,Q1))    -   i. Case of three neighbors Q1, Q2 and Q3        -   g. Let d1=∥P−Q1∥², d2=∥P−Q2∥² and d3=∥P−Q3∥²

${h.\mspace{14mu}{\alpha\left( {P,{Q\; 1}} \right)}} = {\frac{\frac{1}{d\; 1}}{\frac{1}{d\; 1} + \frac{1}{d\; 2} + \frac{1}{d3}} = \frac{d\; 2\; d\; 3}{{d\; 1d\; 2} + {d\; 2d\; 3} + {d\; 1d\; 3}}}$

-   -     i. Approximate α(P,Q1) by applying the division approximation        algorithm with a=d2d3 and b=d1d2+d2d3+d1d3        -   j. Approximate α(P,Q2) by applying the division            approximation algorithm with a=d1d3 and b=d1d2+d2d3+d1d3        -   k. Approximate α(P,Q3) with (2^(n)−α(P,Q1)−α(P,Q2))    -   i. Other considerations        -   l. If d1=0 or d2>2^(n)d1, then treat the situation the same            as if the point has a single neighbor Q1        -   m. If d3>2^(n)d1, then treat the situation the same as if            the point has two neighbors Q1 and Q2        -   n. To be able to support arbitrary large distances with a            limited precision, a quantization procedure could be applied            to d1, d2 and d3

In some embodiments, a fixed-point implementation may be used todetermine lifting scheme updates. For example, fixed pointimplementations may be used in lifting scheme calculations as describedabove, such as in FIGS. 12A and 12B and the associated portions of thespecification for lifting schemes. An example fixed point implementationused in lifting scheme calculations is shown below, wherein updateoperation for P is defined as follows:

${Update}\mspace{11mu}{(P) = \frac{\Sigma_{Q \in {\Delta{(P)}}}\left\lbrack {\left\{ {\alpha\left( {P,Q} \right)} \right\}^{\gamma} \times {w(Q)} \times {D(Q)}} \right\rbrack}{\Sigma_{Q \in {\Delta{(P)}}}\left\lbrack {\left\{ {\alpha\left( {P,Q} \right)} \right\}^{\gamma} \times {w(Q)}} \right\rbrack}}$

-   -   where γ is a parameter usually set to 1 or 2.

In the above equation, division is avoided by leveraging the algorithmdescribed above in regard to fixed point division to approximate theweights

$\frac{\left\{ {\alpha\left( {P,Q} \right)} \right\}^{\gamma} \times {w(Q)}}{\Sigma_{Q \in {\Delta{(P)}}}\left\lbrack {\left\{ {\alpha\left( {P,Q} \right)} \right\}^{\gamma} \times {w(Q)}} \right\rbrack}$

In some embodiments, to reduce the dynamic range of the influenceweights w(Q), use √{square root over (w(Q))} instead of w(Q) and setγ=1.

Thus, the equation becomes:

${{Update}\mspace{11mu}(P)} = \frac{\Sigma_{Q \in {\Delta{(P)}}}\left\lbrack {{\alpha\left( {P,Q} \right)} \times \sqrt{w(Q)} \times {D(Q)}} \right\rbrack}{\Sigma_{Q \in {\Delta{(P)}}}\left\lbrack {{\alpha\left( {P,Q} \right)} \times \sqrt{w(Q)}} \right\rbrack}$

In some embodiments, known fixed-point approximation techniques may beused to compute an approximation of the square root operation.

In some embodiments, a fixed-point implementation may be used todetermine quantized coefficients. For example, fixed pointimplementations may be used in quantization calculations as describedabove, such as in 810 of FIG. 8 and the associated portions of thespecification that discuss quantization. As described above, thequantization procedure leverages the influence weights in order to guidethe quantization process. More precisely, the transform coefficientsassociated with a point P are multiplied by a factor of {w(P)}^(β) (e.g.β=0.5) before applying quantization. An inverse scaling process by thesame factor is applied after inverse quantization on the decoder side.

By choosing β=0.5, the values of b={w(P)}^(0.5) computed during thelifting updates may be re-used.

An example fixed point implementation used in quantization calculationsis shown below, wherein the quantization of a coefficient C with aquantization step q to get a quantized coefficient Ĉ is as follows:

$\hat{C} = \left\{ \begin{matrix}{{floor}\ \left( {\frac{C}{\frac{q}{\left\{ {w(P)} \right\}^{\beta}}} + \delta} \right)} & {C \geq 0} \\{- {{floor}\ \left( {\frac{- c}{\frac{q}{\left\{ {w(P)} \right\}^{\beta}}} + \delta} \right)}} & {C < 0}\end{matrix} \right.$

-   -   Where δ is a parameter that controls the dead zone size (e.g.,        δ=⅓).

The reconstructed coefficient {tilde over (C)} by inverse quantizationis given by:

$\overset{˜}{C} = {\hat{C}\frac{q}{\left\{ {w(P)} \right\}^{\beta}}}$

In a fixed-point implementation in order to avoid such divisionoperations on both the encoder side and the decoder side, theapproximation

$\frac{q}{\left\{ {w(P)} \right\}^{\beta}}$

as described above as an alternative to fixed-point division operationsmay be used with a=q and b={w(P)}^(β).

Point Cloud Attribute Transfer Algorithm

In some embodiments, a point cloud transfer algorithm may be used tominimize distortion between an original point cloud and a reconstructedversion of the original point cloud. A transfer algorithm may be used toevaluate distortion due to the original point cloud and thereconstructed point cloud having points that are in slightly differentpositions. For example, a reconstructed point cloud may have a similarshape as an original point cloud, but may have a.) a different number oftotal points and/or b.) points that are slightly shifted as compared toa corresponding point in the original point cloud. In some embodiments,a point cloud transfer algorithm may allow the attribute values for areconstructed point cloud to be selected such that distortion betweenthe original point cloud and a reconstructed version of the originalpoint cloud is minimized. For example, for an original point cloud, boththe positions of the points and the attribute values of the points areknown. However, for a reconstructed point cloud, the position values maybe known (for example based on a sub-sampling process, K-D tree process,or patch image process as described above). However, attribute valuesfor the reconstructed point cloud may still need to be determined.Accordingly a point cloud transfer algorithm can be used to minimizedistortion by selecting attribute values for the reconstructed pointcloud that minimize distortion.

The distortion from the original point cloud to the reconstructed pointcloud can be determined for a selected attribute value. Likewise thedistortion from the reconstructed point cloud to the original pointcloud can be determined for the selected attribute value for thereconstructed point cloud. In many circumstances, these distortions arenot symmetric. The point cloud transfer algorithm is initialized withtwo errors (E21) and (E12), where E21 is the error from the second orreconstructed point cloud to the original or first point cloud and E12is the error from the first or original point cloud to the second orreconstructed point cloud. For each point in the second point cloud, itis determined whether the point should be assigned the attribute valueof the corresponding point in the original point cloud, or an averageattribute value of the nearest neighbors to the corresponding point inthe original point cloud. The attribute value is selected based on thesmallest error.

Exampled Applications for Point Cloud Compression and Decompression

FIG. 14 illustrates compressed point clouds being used in a 3-Dtelepresence application, according to some embodiments.

In some embodiments, a sensor, such as sensor 102, an encoder, such asencoder 104 or encoder 202, and a decoder, such as decoder 116 ordecoder 220, may be used to communicate point clouds in a 3-Dtelepresence application. For example, a sensor, such as sensor 102, at1402 may capture a 3D image and at 1404, the sensor or a processorassociated with the sensor may perform a 3D reconstruction based onsensed data to generate a point cloud.

At 1406, an encoder such as encoder 104 or 202 may compress the pointcloud and at 1408 the encoder or a post processor may packetize andtransmit the compressed point cloud, via a network 1410. At 1412, thepackets may be received at a destination location that includes adecoder, such as decoder 116 or decoder 220. The decoder may decompressthe point cloud at 1414 and the decompressed point cloud may be renderedat 1416. In some embodiments a 3-D telepresence application may transmitpoint cloud data in real time such that a display at 1416 representsimages being observed at 1402. For example, a camera in a canyon mayallow a remote user to experience walking through a virtual canyon at1416.

FIG. 15 illustrates compressed point clouds being used in a virtualreality (VR) or augmented reality (AR) application, according to someembodiments.

In some embodiments, point clouds may be generated in software (forexample as opposed to being captured by a sensor). For example, at 1502virtual reality or augmented reality content is produced. The virtualreality or augmented reality content may include point cloud data andnon-point cloud data. For example, a non-point cloud character maytraverse a landscape represented by point clouds, as one example. At1504, the point cloud data may be compressed and at 1506 the compressedpoint cloud data and non-point cloud data may be packetized andtransmitted via a network 1508. For example, the virtual reality oraugmented reality content produced at 1502 may be produced at a remoteserver and communicated to a VR or AR content consumer via network 1508.At 1510, the packets may be received and synchronized at the VR or ARconsumer's device. A decoder operating at the VR or AR consumer's devicemay decompress the compressed point cloud at 1512 and the point cloudand non-point cloud data may be rendered in real time, for example in ahead mounted display of the VR or AR consumer's device. In someembodiments, point cloud data may be generated, compressed,decompressed, and rendered responsive to the VR or AR consumermanipulating the head mounted display to look in different directions.

In some embodiments, point cloud compression as described herein may beused in various other applications, such as geographic informationsystems, sports replay broadcasting, museum displays, autonomousnavigation, etc.

Example Computer System

FIG. 16 illustrates an example computer system 1600 that may implementan encoder or decoder or any other ones of the components describedherein, (e.g., any of the components described above with reference toFIGS. 1-15), in accordance with some embodiments. The computer system1600 may be configured to execute any or all of the embodimentsdescribed above. In different embodiments, computer system 1600 may beany of various types of devices, including, but not limited to, apersonal computer system, desktop computer, laptop, notebook, tablet,slate, pad, or netbook computer, mainframe computer system, handheldcomputer, workstation, network computer, a camera, a set top box, amobile device, a consumer device, video game console, handheld videogame device, application server, storage device, a television, a videorecording device, a peripheral device such as a switch, modem, router,or in general any type of computing or electronic device.

Various embodiments of a point cloud encoder or decoder, as describedherein may be executed in one or more computer systems 1600, which mayinteract with various other devices. Note that any component, action, orfunctionality described above with respect to FIGS. 1-15 may beimplemented on one or more computers configured as computer system 1600of FIG. 16, according to various embodiments. In the illustratedembodiment, computer system 1600 includes one or more processors 1610coupled to a system memory 1620 via an input/output (I/O) interface1630. Computer system 1600 further includes a network interface 1640coupled to I/O interface 1630, and one or more input/output devices1650, such as cursor control device 1660, keyboard 1670, and display(s)1680. In some cases, it is contemplated that embodiments may beimplemented using a single instance of computer system 1600, while inother embodiments multiple such systems, or multiple nodes making upcomputer system 1600, may be configured to host different portions orinstances of embodiments. For example, in one embodiment some elementsmay be implemented via one or more nodes of computer system 1600 thatare distinct from those nodes implementing other elements.

In various embodiments, computer system 1600 may be a uniprocessorsystem including one processor 1610, or a multiprocessor systemincluding several processors 1610 (e.g., two, four, eight, or anothersuitable number). Processors 1610 may be any suitable processor capableof executing instructions. For example, in various embodimentsprocessors 1610 may be general-purpose or embedded processorsimplementing any of a variety of instruction set architectures (ISAs),such as the x86, PowerPC, SPARC, or MIPS ISAs, or any other suitableISA. In multiprocessor systems, each of processors 1610 may commonly,but not necessarily, implement the same ISA.

System memory 1620 may be configured to store point cloud compression orpoint cloud decompression program instructions 1622 and/or sensor dataaccessible by processor 1610. In various embodiments, system memory 1620may be implemented using any suitable memory technology, such as staticrandom access memory (SRAM), synchronous dynamic RAM (SDRAM),nonvolatile/Flash-type memory, or any other type of memory. In theillustrated embodiment, program instructions 1622 may be configured toimplement an image sensor control application incorporating any of thefunctionality described above. In some embodiments, program instructionsand/or data may be received, sent or stored upon different types ofcomputer-accessible media or on similar media separate from systemmemory 1620 or computer system 1600. While computer system 1600 isdescribed as implementing the functionality of functional blocks ofprevious Figures, any of the functionality described herein may beimplemented via such a computer system.

In one embodiment, I/O interface 1630 may be configured to coordinateI/O traffic between processor 1610, system memory 1620, and anyperipheral devices in the device, including network interface 1640 orother peripheral interfaces, such as input/output devices 1650. In someembodiments, I/O interface 1630 may perform any necessary protocol,timing or other data transformations to convert data signals from onecomponent (e.g., system memory 1620) into a format suitable for use byanother component (e.g., processor 1610). In some embodiments, I/Ointerface 1630 may include support for devices attached through varioustypes of peripheral buses, such as a variant of the Peripheral ComponentInterconnect (PCI) bus standard or the Universal Serial Bus (USB)standard, for example. In some embodiments, the function of I/Ointerface 1630 may be split into two or more separate components, suchas a north bridge and a south bridge, for example. Also, in someembodiments some or all of the functionality of I/O interface 1630, suchas an interface to system memory 1620, may be incorporated directly intoprocessor 1610.

Network interface 1640 may be configured to allow data to be exchangedbetween computer system 1600 and other devices attached to a network1685 (e.g., carrier or agent devices) or between nodes of computersystem 1600. Network 1685 may in various embodiments include one or morenetworks including but not limited to Local Area Networks (LANs) (e.g.,an Ethernet or corporate network), Wide Area Networks (WANs) (e.g., theInternet), wireless data networks, some other electronic data network,or some combination thereof. In various embodiments, network interface1640 may support communication via wired or wireless general datanetworks, such as any suitable type of Ethernet network, for example;via telecommunications/telephony networks such as analog voice networksor digital fiber communications networks; via storage area networks suchas Fibre Channel SANs, or via any other suitable type of network and/orprotocol.

Input/output devices 1650 may, in some embodiments, include one or moredisplay terminals, keyboards, keypads, touchpads, scanning devices,voice or optical recognition devices, or any other devices suitable forentering or accessing data by one or more computer systems 1600.Multiple input/output devices 1650 may be present in computer system1600 or may be distributed on various nodes of computer system 1600. Insome embodiments, similar input/output devices may be separate fromcomputer system 1600 and may interact with one or more nodes of computersystem 1600 through a wired or wireless connection, such as over networkinterface 1640.

As shown in FIG. 16, memory 1620 may include program instructions 1622,which may be processor-executable to implement any element or actiondescribed above. In one embodiment, the program instructions mayimplement the methods described above. In other embodiments, differentelements and data may be included. Note that data may include any dataor information described above.

Those skilled in the art will appreciate that computer system 1600 ismerely illustrative and is not intended to limit the scope ofembodiments. In particular, the computer system and devices may includeany combination of hardware or software that can perform the indicatedfunctions, including computers, network devices, Internet appliances,PDAs, wireless phones, pagers, etc. Computer system 1600 may also beconnected to other devices that are not illustrated, or instead mayoperate as a stand-alone system. In addition, the functionality providedby the illustrated components may in some embodiments be combined infewer components or distributed in additional components. Similarly, insome embodiments, the functionality of some of the illustratedcomponents may not be provided and/or other additional functionality maybe available.

Those skilled in the art will also appreciate that, while various itemsare illustrated as being stored in memory or on storage while beingused, these items or portions of them may be transferred between memoryand other storage devices for purposes of memory management and dataintegrity. Alternatively, in other embodiments some or all of thesoftware components may execute in memory on another device andcommunicate with the illustrated computer system via inter-computercommunication. Some or all of the system components or data structuresmay also be stored (e.g., as instructions or structured data) on acomputer-accessible medium or a portable article to be read by anappropriate drive, various examples of which are described above. Insome embodiments, instructions stored on a computer-accessible mediumseparate from computer system 1600 may be transmitted to computer system1600 via transmission media or signals such as electrical,electromagnetic, or digital signals, conveyed via a communication mediumsuch as a network and/or a wireless link. Various embodiments mayfurther include receiving, sending or storing instructions and/or dataimplemented in accordance with the foregoing description upon acomputer-accessible medium. Generally speaking, a computer-accessiblemedium may include a non-transitory, computer-readable storage medium ormemory medium such as magnetic or optical media, e.g., disk orDVD/CD-ROM, volatile or non-volatile media such as RAM (e.g. SDRAM, DDR,RDRAM, SRAM, etc.), ROM, etc. In some embodiments, a computer-accessiblemedium may include transmission media or signals such as electrical,electromagnetic, or digital signals, conveyed via a communication mediumsuch as network and/or a wireless link

The methods described herein may be implemented in software, hardware,or a combination thereof, in different embodiments. In addition, theorder of the blocks of the methods may be changed, and various elementsmay be added, reordered, combined, omitted, modified, etc. Variousmodifications and changes may be made as would be obvious to a personskilled in the art having the benefit of this disclosure. The variousembodiments described herein are meant to be illustrative and notlimiting. Many variations, modifications, additions, and improvementsare possible. Accordingly, plural instances may be provided forcomponents described herein as a single instance. Boundaries betweenvarious components, operations and data stores are somewhat arbitrary,and particular operations are illustrated in the context of specificillustrative configurations. Other allocations of functionality areenvisioned and may fall within the scope of claims that follow. Finally,structures and functionality presented as discrete components in theexample configurations may be implemented as a combined structure orcomponent. These and other variations, modifications, additions, andimprovements may fall within the scope of embodiments as defined in theclaims that follow.

What is claimed is:
 1. A system, comprising: an encoder configured to:organize a plurality of points of a point cloud into an order accordingto a space filling curve based on respective spatial positions of theplurality of points of the point cloud in 3D space; assign an attributevalue to at least one point of a first level of detail for the pointcloud, wherein points of the point cloud to be included in the firstlevel of detail are selected based, at least in part, on theirrespective positions in the order according to the space filling curve;for respective points of the other points of the first level of detail,and points of one or more additional levels of detail for the pointcloud, determine a predicted attribute value for the respective pointbased on predicted or assigned attributes values for neighboring pointsin a same level of detail as the point for which an attribute value isbeing predicted, wherein points to be included in the one or moreadditional levels of detail are selected based, at least in part, ontheir respective positions in the order according to the space fillingcurve, and wherein the neighboring points used to determine thepredicted attribute value for the respective point for which anattribute value is being predicted are selected based, at least in part,on their respective positions in the space filling relative to therespective point for which an attribute value is being predicted; forrespective points of the other points of the first level of detail, andthe points of the one or more additional levels of detail, determine anattribute correction value for the respective point, based on comparinga predicted attribute value for the respective point to attributeinformation for the point included in the point cloud being encoded thatcorresponds with the respective point; apply an update operation tosmooth the attribute correction values, wherein the update operationtakes into account relative influences of the attributes of the pointsof a given level of detail on attribute values of points included inother levels of detail; and encode the assigned attribute value and theupdated attribute correction values for first level of detail or the oneor more additional levels of detail.
 2. The system of claim 1, whereinthe first level of detail includes fewer points of the point cloud thanthe one or more additional levels of detail, wherein the one or moreadditional levels of detail are respective subsequent levels of detailsthat further refine the encoded point cloud by adding more points to apreceding level of detail, and wherein the encoder is configured to:encode the updated attribute correction values for preceding ones of thelevels of detail before encoding updated attribute correction values forone or more subsequent ones of the levels of detail.
 3. The system ofclaim 1, wherein the encoder is configured to determine predictedattribute values for first level of detail or a given one of the one ormore additional levels of detail prior to determining which points ofthe point cloud are to be included in one or more other ones of the oneor more additional levels of detail.
 4. The system of claim 1, whereinto select the points to include in the first level of detail or thepoints to include in the one or more additional levels of detail based,at least in part, on the order according to the space filling curve, theencoder is configured to: assign index values to respective ones of thepoints of the point cloud based on respective positions of therespective points in the order according to the space filling curve; andfor each point having a given index value: identify a set of neighboringpoints having index values within a search range of the given indexvalue; determine respective distances between the point with the givenvalue and respective ones of the neighboring points having index valueswithin the search range; and include neighboring points in a given levelof detail being evaluated if the determined respective distances areless than an inclusion distance for the given level of detail beingevaluated.
 5. The system of claim 4, wherein the search range, aninclusion distance for the first level of detail, and an inclusiondistance ratio between levels of detail are user defined parameters, andwherein the inclusion distance for the given level of detail beingevaluated is determined based on an inclusion distance for a precedinglevel of detail and the inclusion distance ratio between levels ofdetail.
 6. The system of claim 5, wherein to identify the neighboringpoints used to determine a predicted value for a respective point forwhich an attribute value is being predicted, the encoder is configuredto: identify a set of neighboring points having index values within asearch range of a given index value for the point for which an attributevalue is being predicted.
 7. The system of claim 5, wherein the searchrange used for identifying the set of neighboring points for determininga predicted attribute value and the search range used for selectingpoints to include in a given level of detail are different searchranges.
 8. The system of claim 6, wherein the search range used foridentifying the set of neighboring points for determining a predictedattribute value is a user defined parameter; and wherein a number ofnearest neighboring points to be used for determining the predictedattribute value is a user defined parameter.
 9. The system of claim 1,wherein the order according to the space filling curve organizes thepoints of the point cloud according to a Morton order.
 10. The system ofclaim 1, wherein to encode the updated attribute correction values forthe first level of detail or for the one or more additional levels ofdetail, the encoder is configured to: map an N-bit value for an updatedattribute correction value to an M-bit code word based on an entry forthe N-bit value in a look-up table of the encoder, wherein M is lessthan N, and wherein a decoder recreates a look-up table that correspondsto the look-up table of the encoder; and encode, via an adaptive binaryarithmetic encoding component of the encoder, the M-bit code word inplace of the N-bit value for the updated attribute correction value. 11.The system of claim 10, wherein to encode the updated attributecorrection values for the first level of detail or the one or moreadditional levels of detail, the encoder is configured to: determinewhether the updated attribute correction value is zero or not, and ifthe attribute correction value is zero, encode, via the adaptive binaryarithmetic encoding component of the encoder, a value of zero for theupdated attribute correction value instead of encoding an M-bit codeword.
 12. The system of claim 10, wherein to encode the updatedattribute correction values for the first level of detail or the one ormore additional levels of detail, the encoder is configured to:determine whether the updated attribute correction value is equal to orgreater than a maximum updated attribute correction value included inthe look-up table of the encoder or not, and if the attribute correctionvalue is equal to or greater than the maximum updated attributecorrection value included in the look-up table of the encoder: encode,via the adaptive binary arithmetic encoder, an M-bit word correspondingto the maximum updated attribute correction value included in thelook-up table of the encoder; and encode, via an additional encodingcomponent of the encoder, a value corresponding to an amount by which,if any, the updated attribute correction value exceeds the maximumupdated attribute correction value included in the look-up table of theencoder.
 13. A method, comprising: determining an order for a pluralityof points of a point cloud according to a space filling curve based onrespective spatial positions of the points of the point cloud in 3Dspace; determining predicted attribute values for points of the pointcloud included in a first level of detail or one or more additionallevels of detail based on neighboring points in a same level of detailas the point for which a predicted attribute value is being determined,wherein points to be included in the first level of detail and the oneor more additional levels of detail are selected based, at least inpart, on their respective positions in the order according to the spacefilling curve, and wherein the neighboring points used to determine thepredicted attribute value, for the point for which an attribute value isbeing predicted, are selected based, at least in part, on theirrespective positions in the order according to the space filling curverelative to the point for which an attribute value is being predicted;determining attribute correction values for the points of the pointcloud included in the first level of detail or the one or moreadditional levels of detail based on comparing the determined predictedattribute values for the points to attribute values of correspondingpoints of the point cloud; applying an update operation to smooth theattribute correction values, wherein the update operation takes intoaccount relative influences of the attributes of the points of a givenlevel of detail on attribute values of points included in other levelsof detail; and encoding the updated attribute correction values.
 14. Themethod of claim 13, wherein said determining predicted attribute values,said determining attribute correction values, said applying the updateoperation, and said encoding is performed according to a bottom-upapproach, wherein levels of detail comprising fewer points are processedbefore levels of detail that add additional point to preceding levels ofdetails comprising fewer points.
 15. The method of claim 13, whereinassigning points to one or more higher levels of detail is performedconcurrently with predicting attribute values for one or more lowerlevels of detail.
 16. The method of claim 13, wherein said encoding theupdated attribute correction values comprises: mapping an N-bit valuefor an updated attribute correction value to an M-bit code word based onan entry for the N-bit value in a look-up table of an encoder performingthe encoding, wherein M is less than N, and wherein a decoder recreatesa look-up table that corresponds to the look-up table of the encoder;and encoding, via an adaptive binary arithmetic encoding component ofthe encoder, the M-bit code word in place of the N-bit value for theupdated attribute correction value.
 17. The method of claim 16, furthercomprising: determining the updated attribute correction value is zeroand encoding a value of zero instead of the M-bit code word; ordetermining the attribute correction value is greater than a maximumupdated attribute correction value included in the look-up table of theencoder, and: encoding, via the adaptive binary arithmetic encodingcomponent of the encoder, an M-bit code work corresponding to themaximum updated attribute correction value included in the look up tableof the encoder; and encoding, via a Golomb encoding component of theencoder, a value corresponding to an amount by which the updatedattribute correction value exceeds the maximum updated attributecorrection value included in the look-up table.
 18. A system,comprising: a decoder configured to: receive compressed attributeinformation for a point cloud comprising at least one assigned attributevalue for at least one point of a first level of detail of the pointcloud and data indicating attribute correction values for attributes ofother points of the point cloud in the first level of detail or in oneor more additional levels of detail; determine attribute information fora decompressed point cloud comprising the first level of detail or oneor more of the one or more additional levels of detail, wherein todetermine the attribute information for the decompressed point cloud,the decoder is configured to: predict attribute values for points of thepoint cloud included in the first level of detail or the one or moreadditional levels of detail based on neighboring points in a same levelof detail as the point for which a predicted attribute value is beingdetermined, wherein the neighboring points used to determine thepredicted attribute value for the point for which an attribute value isbeing predicted are selected based, at least in part, on theirrespective positions in a space filling curve relative to the point forwhich an attribute value is being predicted; and update the predictedattribute values based on the attribute correction values included inthe received compressed attribute information.
 19. The system of claim18, wherein the encoder is configured to: determine, for respective onesof the attribute correction values, whether the respective attributecorrection value is zero or not; and if the attribute correction valueis not zero, determine an N-bit attribute correction value based on anM-bit code word, wherein the received compressed attribute informationcomprises M-bit code words to represent the attribute correction values,wherein N is greater than M, and wherein the decoder maintains a look-uptable mapping M-bit code words to N-bit attribute correction values. 20.The system of claim 19, wherein the decoder is further configured to:determine, for the respective ones of the attribute correction values,whether the respective attribute correction value corresponds to agreatest M-bit code word included in the look-up table; and if theattribute correction value corresponds to the maximum greatest M-bitcode word included in the look-up table: determine a correspondingattribute correction value included in the look-up table for the M-bitcode word; decode, via a Golomb decoding component of the decoder, anadditional value to be added to the maximum attribute correction valueincluded in the look-up table; and add the additional value to thecorresponding attribute correction value included in the look-up tablefor the M-bit code word to result in the respective attribute correctionvalue.